You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Sophie Blee-Goldman <so...@confluent.io> on 2020/05/13 02:27:54 UTC

[DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Hey all,

I'd like to kick off discussion on KIP-613 which aims to add end-to-end
latency metrics to Streams. Please take a look:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams

Cheers,
Sophie

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Hey all,

I'd like to make one adjustment to the current proposal and go back
to the earlier proposal to report the task-level e2e latency metric at the
source as well as the sink. I feel this will give users a more complete
picture of the e2e latency if they only turn on INFO level metrics.

I also want to quickly clarify one point, which is that these metrics will
still
be reported at the end of a subtopology that includes no sink. The latency
will just be recorded at whatever the terminal operator happens to be.

Please let me know if you have any questions or concerns.
Thanks!
Sophie

On Mon, May 18, 2020 at 9:32 AM Guozhang Wang <wa...@gmail.com> wrote:

> Thanks Sophie, I'm +1 on the updated KIP.
>
> On Mon, May 18, 2020 at 8:32 AM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
>
> > @Bruno
> > No, this would be after inserting in the cache/state store/etc --
> basically
> > once
> > all operations have completed such that the record would be present if
> > queried after that time.
> >
> > @Guozhang
> > I'm glad you suggested the TRACE level. It seems we've been
> > heading in that direction for a while so I'm happy to add it as part of
> > this KIP. Especially if it unblocks the question of whether to update the
> > system time (for now).
> >
> > On Mon, May 18, 2020 at 1:09 AM Bruno Cadonna <br...@confluent.io>
> wrote:
> >
> > > Hi Sophie,
> > >
> > > Just a question for my understanding. When you say " t_A be the time
> > > when it is finished being processed by the aggregator node", this
> > > means before caching (or putting into the state store if caching is
> > > disabled), right?
> > >
> > > Best,
> > > Bruno
> > >
> > > On Sat, May 16, 2020 at 6:57 PM Guozhang Wang <wa...@gmail.com>
> > wrote:
> > > >
> > > > We do have a big JIRA for making sensor recording to be sufficient.
> But
> > > it
> > > > is a bit hard to try to reduce the num.system clock calls in general
> > (but
> > > > admittedly, in nowadays OS they are supposed to be very cheap as
> well),
> > > > I've just recently merged a small PR trying to make less
> > > > SystemTime.currentMilliseconds calls whenever possible, but it is
> hard
> > to
> > > > enforce that for all callers.
> > > >
> > > > Guozhang
> > > >
> > > > On Sat, May 16, 2020 at 9:46 AM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > > > FWIW, I’d be fine with TRACE, if it makes the processor-level
> metrics
> > > more
> > > > > palatable to everyone.
> > > > >
> > > > > Ultimately, I think this is just kicking the can down the road, and
> > we
> > > > > should really just refactor our sensors/metrics to be lower
> overhead.
> > > But
> > > > > that’s too much work to consider right before the release so for
> now,
> > > we
> > > > > can kick that can just a little farther.
> > > > >
> > > > > Thanks for proposing a solution, Guozhang!
> > > > >
> > > > > -John
> > > > >
> > > > > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> > > > > > Cool, that's good points -- we should not let our current metrics
> > > > > proposal
> > > > > > to depend on future roadmaps, since they can still be removed
> later
> > > if
> > > > > ever
> > > > > > becoming not valuable after all.
> > > > > >
> > > > > > On the other hand, what do you think about introducing TRACE
> level
> > > > > metrics
> > > > > > recording and make the the processor-node level metrics on TRACE?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <
> > > sophie@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yeah, I had proposed to replace the "source + sink" task-level
> > > metric
> > > > > > > with "sink only" task-level metrics plus the stateful
> node-level
> > > ones.
> > > > > > >
> > > > > > > I completely agree that this proposal should and would look
> > > different
> > > > > > > once we complete those two proposals, ie decouple caching and
> > > > > > > consolidate suppression. But as it is right now, caching and
> > > > > suppression
> > > > > > > cause the actual time-of-processing of records to vary greatly
> > > > > throughout
> > > > > > > a subtopology such that the delta between a source node and an
> > > > > intermediate
> > > > > > > one is potentially quite large.
> > > > > > >
> > > > > > > For this exact reason I would argue that this metric is still
> > > valuable
> > > > > at
> > > > > > > the node-level
> > > > > > >  even if the "current time" is actually only updated once for a
> > > task.
> > > > > > > Obviously this
> > > > > > > would not convey any information about the actual processing
> > > latency,
> > > > > but
> > > > > > > as you
> > > > > > > stated, this is likely to be a small delta on the scale of
> > > > > milliseconds.
> > > > > > > But the latency
> > > > > > > due to a suppression can be on the order of minutes, hours, or
> > even
> > > > > days!
> > > > > > > If
> > > > > > > there is more than one suppression or caching in a subtopology,
> > the
> > > > > source
> > > > > > > and sink
> > > > > > > e2e latency could be wildly misrepresentative of the e2e
> latency
> > > of a
> > > > > node
> > > > > > > in the middle.
> > > > > > >
> > > > > > > Anyways, I removed this last point from the current KIP
> document
> > > > > because I
> > > > > > > felt
> > > > > > > it was ultimately an implementation detail. But the larger
> point
> > I
> > > > > would
> > > > > > > argue is that:
> > > > > > > if we are truly worried about the performance hit of fetching
> the
> > > > > system
> > > > > > > time at every
> > > > > > > node, we should just not fetch the system time at every node
> > > instead of
> > > > > > > dropping the
> > > > > > > metric altogether. It may be slightly off but only on the order
> > of
> > > ms
> > > > > and
> > > > > > > not on the order
> > > > > > > of hours, and it's still more useful than no metric at all
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Sophie,
> > > > > > > >
> > > > > > > > I think your motivation makes sense, in fact before KIP-444
> we
> > > used
> > > > > to
> > > > > > > have
> > > > > > > > processor-node-level process latency as DEBUG which would
> > > > > holistically
> > > > > > > > address the motivation but we removed it since it is indeed
> > quite
> > > > > > > expensive
> > > > > > > > (I do not have concrete numbers at hand, but basically you'd
> > > have to
> > > > > call
> > > > > > > > system-wall-clock time a couple of time for each record, each
> > > > > processor
> > > > > > > > node), and in the long run I feel such scenarios would be
> less
> > > > > common if
> > > > > > > we
> > > > > > > > 1) decouple caching from emitting and 2) always put
> suppression
> > > right
> > > > > > > > before sink only, and that's also why I've proposed to remove
> > > them in
> > > > > > > > KIP-444.
> > > > > > > >
> > > > > > > > I missed an update in your wiki that right now you've updated
> > the
> > > > > > > > task-level metric to be recorded at the sink node, and
> thought
> > > it is
> > > > > > > still
> > > > > > > > at the source node, but assuming it is still the case, then
> > > > > practically
> > > > > > > any
> > > > > > > > state store's staleness of the task could roughly be measured
> > as
> > > that
> > > > > > > > task-level staleness plus a small delta (usually tens of
> > mills),
> > > and
> > > > > the
> > > > > > > > down stream task B's staleness - up stream task A's staleness
> > > would
> > > > > > > capture
> > > > > > > > the suppression effect at the end of A plus the intermediate
> > > topic
> > > > > > > > producing -> consuming latency, which usually would not
> require
> > > to be
> > > > > > > > distinguished.
> > > > > > > >
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <
> > > > > sophie@confluent.io
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Guozhang,
> > > > > > > > >
> > > > > > > > > Thanks for the response. I meant to ask this earlier and
> > > forgot,
> > > > > but do
> > > > > > > > you
> > > > > > > > > have any data or benchmarking results on hand for the
> > > performance
> > > > > hit
> > > > > > > we
> > > > > > > > > took due to the metrics? Or maybe better yet, on the
> > > performance we
> > > > > > > > gained
> > > > > > > > > back by dropping some metrics? I think it would help us to
> > > better
> > > > > > > > > understand
> > > > > > > > > the tradeoffs with some concrete numbers to compare.
> > > > > > > > >
> > > > > > > > > I guess I'm inclined to want to include these metrics at
> the
> > > > > operator
> > > > > > > > level
> > > > > > > > > on
> > > > > > > > > DEBUG since, in my mind, that's exactly what DEBUG is for:
> > > > > performance-
> > > > > > > > > focused users will presumably run at INFO level and even
> > those
> > > who
> > > > > do
> > > > > > > > > use DEBUG may only do so temporarily, when they actually
> have
> > > > > something
> > > > > > > > > to debug.
> > > > > > > > >
> > > > > > > > > Of course, I'm not running a Streams app myself so all this
> > is
> > > just
> > > > > > > > > conjecture.
> > > > > > > > > But it seems reasonable to interpret the two metrics levels
> > as
> > > > > "minimal
> > > > > > > > > useful
> > > > > > > > > information" and "all information that provides insight
> into
> > > the
> > > > > > > system"
> > > > > > > > >
> > > > > > > > > If we can agree there, the question becomes "does this
> > provide
> > > any
> > > > > > > > > additional
> > > > > > > > > insight?" I would still argue that it does. To give a
> > concrete
> > > > > example,
> > > > > > > > > let's say
> > > > > > > > > the user has a basic
> > > > > > > > >
> > > > > > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > > > > > > >
> > > > > > > > > subtopology, the application performing IQ on the
> > transformer's
> > > > > state.
> > > > > > > > The
> > > > > > > > > user
> > > > > > > > > wants to know how long it's taking records to be reflected
> in
> > > the
> > > > > > > results
> > > > > > > > > of a query.
> > > > > > > > > With only task-level metrics, they can only guess based on
> > the
> > > e2e
> > > > > > > > latency
> > > > > > > > > of the
> > > > > > > > > previous task and of this one. But neither of those is a
> good
> > > > > > > > > representation of the
> > > > > > > > > time it will take a record to be processed by the
> > transformer:
> > > the
> > > > > > > > previous
> > > > > > > > > task's
> > > > > > > > > latency will not include the time for the record to go
> > through
> > > the
> > > > > > > > > repartitioning,
> > > > > > > > > and the current task's latency is measured after a
> > > suppression. The
> > > > > > > > > suppression
> > > > > > > > > itself will introduce significant latency and doesn't
> provide
> > > an
> > > > > > > accurate
> > > > > > > > > estimate.
> > > > > > > > > Caching would also do what it does best, messing with our
> > > > > understanding
> > > > > > > > of
> > > > > > > > > time :)
> > > > > > > > >
> > > > > > > > > It also means you can't measure the consumption latency, so
> > > there
> > > > > would
> > > > > > > > be
> > > > > > > > > no way to tell if the e2e latency is so high because the
> very
> > > first
> > > > > > > > > subtopology is
> > > > > > > > > extremely slow, or the record was severely delayed in
> > reaching
> > > the
> > > > > > > input
> > > > > > > > > topic.
> > > > > > > > > I guess my earlier proposal to include both source and sink
> > > metrics
> > > > > > > could
> > > > > > > > > address this issue, but it seems to solve the problem more
> > > > > holistically
> > > > > > > > --
> > > > > > > > > and for
> > > > > > > > > more complex subtopologies -- by including metrics at the
> > > processor
> > > > > > > level
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey folks,
> > > > > > > > > >
> > > > > > > > > > I want to make a final fight on processor-node-level
> > metrics
> > > :)
> > > > > Post
> > > > > > > > > > KIP-444 we've actually removed a lot of node-level
> metrics
> > > since
> > > > > it
> > > > > > > is
> > > > > > > > > too
> > > > > > > > > > expensive to record at that level and their values are
> not
> > > proven
> > > > > > > worth
> > > > > > > > > it.
> > > > > > > > > > Again I'd use the "we have to either enable all or none
> of
> > > DEBUG
> > > > > > > > metrics"
> > > > > > > > > > card, e.g. if we want to look into per-task
> process-latency
> > > > > which is
> > > > > > > > > DEBUG,
> > > > > > > > > > we'd have to record all process-node / state-store /
> cache
> > > level
> > > > > > > > metrics.
> > > > > > > > > > On the other hand, within a task we usually do not have a
> > > lot of
> > > > > > > > stateful
> > > > > > > > > > operator nodes and if we want to find if certain stateful
> > > nodes
> > > > > are
> > > > > > > the
> > > > > > > > > > bottleneck of process latency, that is usually
> discoverable
> > > from
> > > > > the
> > > > > > > > > > state-store level read / write latencies already. So I'd
> > > imagine
> > > > > that
> > > > > > > > > > suppose there's a sub-topology with multiple state stores
> > in
> > > it,
> > > > > if
> > > > > > > we
> > > > > > > > > > found the task-level process latency is high, we can
> still
> > > tell
> > > > > which
> > > > > > > > > state
> > > > > > > > > > stores within it is the main bottleneck from the state
> > store
> > > > > read /
> > > > > > > > write
> > > > > > > > > > latency (I'm assuming the IO latency is always dominant
> > > compared
> > > > > with
> > > > > > > > > > others).
> > > > > > > > > >
> > > > > > > > > > Guozhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > > > > > > sophie@confluent.io
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > @Matthias
> > > > > > > > > > > > And that's what he says about the 95th percentile!
> > > Imagine
> > > > > what
> > > > > > > he
> > > > > > > > > > would
> > > > > > > > > > > say about the 50th :P
> > > > > > > > > > >
> > > > > > > > > > > I should have kept watching another 10 minutes. He gets
> > > around
> > > > > to
> > > > > > > > > > covering
> > > > > > > > > > > the 50th, and let's
> > > > > > > > > > > just say he is not a fan:
> > > https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <
> > > > > vvcephei@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks, Sophie!
> > > > > > > > > > > >
> > > > > > > > > > > > I think this makes perfect sense. It will be much
> more
> > > > > intuitive
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > > the metric for the stated motivation this way.  I’d
> be
> > in
> > > > > favor
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > proposal after this update.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again for taking this on,
> > > > > > > > > > > > -John
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman
> > > wrote:
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to clarify/modify one aspect of this KIP,
> > > which
> > > > > is to
> > > > > > > > > record
> > > > > > > > > > > the
> > > > > > > > > > > > > staleness
> > > > > > > > > > > > > at the *completion* of the record's processing by
> the
> > > > > > > > operator/task
> > > > > > > > > > in
> > > > > > > > > > > > > question,
> > > > > > > > > > > > > rather than on the intake. The task-level metrics
> > will
> > > be
> > > > > > > > recorded
> > > > > > > > > at
> > > > > > > > > > > the
> > > > > > > > > > > > > sink
> > > > > > > > > > > > > node instead of at the source, and the
> operator-level
> > > > > metrics
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > recorded
> > > > > > > > > > > > > at the end of the operation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The stated purpose and intended usefulness of this
> > KIP
> > > is
> > > > > to
> > > > > > > give
> > > > > > > > > > > users a
> > > > > > > > > > > > > way
> > > > > > > > > > > > > to gauge roughly how long it takes for a record to
> be
> > > > > reflected
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > "results",
> > > > > > > > > > > > > whether these results are being read from an output
> > > topic
> > > > > or
> > > > > > > > > through
> > > > > > > > > > > IQ.
> > > > > > > > > > > > To
> > > > > > > > > > > > > take the IQ example, the results of a record are
> > > obviously
> > > > > not
> > > > > > > > > > visible
> > > > > > > > > > > > until
> > > > > > > > > > > > > *after* that node has finished processing it. The
> > > > > staleness,
> > > > > > > > while
> > > > > > > > > > > still
> > > > > > > > > > > > > potentially
> > > > > > > > > > > > > useful as it can impact the *way* a record is
> > > processed in
> > > > > a
> > > > > > > > > stateful
> > > > > > > > > > > and
> > > > > > > > > > > > > time-
> > > > > > > > > > > > > dependent operator, is not part of the problem this
> > KIP
> > > > > > > > > specifically
> > > > > > > > > > > set
> > > > > > > > > > > > out
> > > > > > > > > > > > > to solve.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In light of this I think it's appropriate to revert
> > the
> > > > > name
> > > > > > > > change
> > > > > > > > > > > back
> > > > > > > > > > > > to
> > > > > > > > > > > > > include
> > > > > > > > > > > > > latency, since "staleness" as described above only
> > > makes
> > > > > sense
> > > > > > > > when
> > > > > > > > > > > > > measuring
> > > > > > > > > > > > > relative to the arrival of a record at a task/node.
> > I'd
> > > > > propose
> > > > > > > > to
> > > > > > > > > > save
> > > > > > > > > > > > the
> > > > > > > > > > > > > term
> > > > > > > > > > > > > "staleness" for that particular meaning, and adopt
> > > > > Matthias's
> > > > > > > > > > > suggestion
> > > > > > > > > > > > of
> > > > > > > > > > > > > "record-e2e-latency" for this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for hanging in there all. Please let me know
> > if
> > > you
> > > > > have
> > > > > > > > any
> > > > > > > > > > > > > concerns about
> > > > > > > > > > > > > this change!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sophie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > > > > > > mjsax@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I am also happy with max/min/99/90. And I buy
> your
> > > naming
> > > > > > > > > argument
> > > > > > > > > > > > about
> > > > > > > > > > > > > > staleness vs latency.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Matthias
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > > > > > > Hey Sophie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie
> > > Blee-Goldman <
> > > > > > > > > > > > > > sophie@confluent.io>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> @Matthias
> > > > > > > > > > > > > > >> Regarding tracking the 50th percentile, I'll
> > refer
> > > > > you to
> > > > > > > > the
> > > > > > > > > > 4:53
> > > > > > > > > > > > mark
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > >> the video
> > > > > > > > > > > > > > >> *you* linked:
> > https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> And that's what he says about the 95th
> > percentile!
> > > > > Imagine
> > > > > > > > > what
> > > > > > > > > > he
> > > > > > > > > > > > would
> > > > > > > > > > > > > > >> say about
> > > > > > > > > > > > > > >> the 50th :P
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> But seriously, since we can't seem to agree
> that
> > > the
> > > > > mean
> > > > > > > or
> > > > > > > > > > 50th
> > > > > > > > > > > > > > >> percentile is actually
> > > > > > > > > > > > > > >> useful I'm inclined to resurrect my original
> > > proposal,
> > > > > > > > > neither.
> > > > > > > > > > > But
> > > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > >> that's a good
> > > > > > > > > > > > > > >> argument against the 75th, which I admittedly
> > > chose
> > > > > > > somewhat
> > > > > > > > > > > > > > arbitrarily as
> > > > > > > > > > > > > > >> an
> > > > > > > > > > > > > > >> intermediate between the 50th and the higher
> > > > > percentiles.
> > > > > > > > How
> > > > > > > > > > > about:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> -max
> > > > > > > > > > > > > > >> -p99
> > > > > > > > > > > > > > >> -p90
> > > > > > > > > > > > > > >> -min
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> with p50/mean still up for debate if anyone
> > feels
> > > > > strongly
> > > > > > > > for
> > > > > > > > > > > > either of
> > > > > > > > > > > > > > >> them.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Regarding the name, I've already flip-flopped
> on
> > > this
> > > > > so
> > > > > > > I'm
> > > > > > > > > > > > definitely
> > > > > > > > > > > > > > >> still open to
> > > > > > > > > > > > > > >> further arguments. But the reason for changing
> > it
> > > from
> > > > > > > > > > > > > > end-to-end-latency
> > > > > > > > > > > > > > >> (which
> > > > > > > > > > > > > > >> is similar to what you propose) is that this
> > > metric
> > > > > > > > > technically
> > > > > > > > > > > > reflects
> > > > > > > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > > > > > > >> the record is when it's *received* by the
> > > operator,
> > > > > not
> > > > > > > when
> > > > > > > > > > it's
> > > > > > > > > > > > > > processed
> > > > > > > > > > > > > > >> by the operator.
> > > > > > > > > > > > > > >> It seemed like there was the potential for
> > > confusion
> > > > > that
> > > > > > > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > > > > > > >> represent the latency from the event creation
> to
> > > the
> > > > > time
> > > > > > > > the
> > > > > > > > > > > > processor
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > >> done
> > > > > > > > > > > > > > >> processing it.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> @John
> > > > > > > > > > > > > > >> I'd rather err on the side of "not-enough"
> > > metrics as
> > > > > we
> > > > > > > can
> > > > > > > > > > > always
> > > > > > > > > > > > add
> > > > > > > > > > > > > > >> this to the
> > > > > > > > > > > > > > >> stateless metrics later on. If we decide to
> > > measure
> > > > > the
> > > > > > > time
> > > > > > > > > at
> > > > > > > > > > > > every
> > > > > > > > > > > > > > node
> > > > > > > > > > > > > > >> and don't
> > > > > > > > > > > > > > >> find any evidence of a serious performance
> > > impact, and
> > > > > > > users
> > > > > > > > > > > > indicate
> > > > > > > > > > > > > > they
> > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > >> like to see this metric at all nodes, then we
> > can
> > > > > easily
> > > > > > > > start
> > > > > > > > > > > > reporting
> > > > > > > > > > > > > > >> them as well.
> > > > > > > > > > > > > > >> WDYT?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> That said, sink nodes seem like a reasonable
> > > > > exception to
> > > > > > > > the
> > > > > > > > > > > rule.
> > > > > > > > > > > > > > >> Obviously users
> > > > > > > > > > > > > > >> should be able to detect the time when the
> > record
> > > > > reaches
> > > > > > > > the
> > > > > > > > > > > output
> > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > >> but that
> > > > > > > > > > > > > > >> still leaves a gap in understanding how long
> the
> > > > > > > production
> > > > > > > > > > > latency
> > > > > > > > > > > > was.
> > > > > > > > > > > > > > >> This mirrors
> > > > > > > > > > > > > > >> the consumption latency that is exposed by the
> > > > > task-level
> > > > > > > > > > metrics,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > >> are measured
> > > > > > > > > > > > > > >> at the source node. For good symmetry what if
> we
> > > > > actually
> > > > > > > > > expose
> > > > > > > > > > > > both
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> source
> > > > > > > > > > > > > > >> and sink latency at the task-level? ie report
> > both
> > > > > sets of
> > > > > > > > > > > > statistical
> > > > > > > > > > > > > > >> measurements with
> > > > > > > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> @Bill
> > > > > > > > > > > > > > >> Thanks for the comment about regarding the
> min!
> > I
> > > > > hadn't
> > > > > > > > > > > considered
> > > > > > > > > > > > that
> > > > > > > > > > > > > > >> and it's
> > > > > > > > > > > > > > >> quite useful to think about how and what is
> > useful
> > > > > from a
> > > > > > > > > users
> > > > > > > > > > > > point of
> > > > > > > > > > > > > > >> view.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Regarding your second. point, I'm inclined to
> > > leave
> > > > > that
> > > > > > > as
> > > > > > > > an
> > > > > > > > > > > > > > >> implementation detail
> > > > > > > > > > > > > > >> but my take would be that the user should be
> > > allowed
> > > > > to
> > > > > > > > > control
> > > > > > > > > > > the
> > > > > > > > > > > > > > record
> > > > > > > > > > > > > > >> timestamp
> > > > > > > > > > > > > > >> used for this with the timestamp extractor. My
> > > > > impression
> > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > > users
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > >> often embed
> > > > > > > > > > > > > > >> the actual event time in the payload for
> > whatever
> > > > > reason,
> > > > > > > > and
> > > > > > > > > > this
> > > > > > > > > > > > > > >> represents the "true"
> > > > > > > > > > > > > > >> timestamp as far as the Streams topology is
> > > concerned.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > > > > > > bbejeck@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a
> > useful
> > > > > metric
> > > > > > > to
> > > > > > > > > > add.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Regarding tracking min, I  think it could be
> > > > > valuable for
> > > > > > > > > users
> > > > > > > > > > > to
> > > > > > > > > > > > > > >> discern
> > > > > > > > > > > > > > >>> which part of their topologies are more
> > efficient
> > > > > since
> > > > > > > > this
> > > > > > > > > > is a
> > > > > > > > > > > > > > >>> task-level metric.  I realize everyone seems
> to
> > > be on
> > > > > > > board
> > > > > > > > > > with
> > > > > > > > > > > > > > >> including
> > > > > > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on
> > > this
> > > > > topic
> > > > > > > > > should
> > > > > > > > > > > we
> > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> I do have a question regarding the
> calculation
> > of
> > > > > > > > staleness.
> > > > > > > > > > > > > > >>> Is there going to be a consideration for
> > > timestamp
> > > > > > > > > extractors?
> > > > > > > > > > > > Users
> > > > > > > > > > > > > > >> could
> > > > > > > > > > > > > > >>> prefer to use a timestamp embedded in the
> > > payload,
> > > > > and it
> > > > > > > > > could
> > > > > > > > > > > > skew
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >>> measurements.
> > > > > > > > > > > > > > >>> I was wondering if we should specify in the
> KIP
> > > if
> > > > > > > setting
> > > > > > > > > the
> > > > > > > > > > > > arrival
> > > > > > > > > > > > > > >> time
> > > > > > > > > > > > > > >>> is always going to come from the record
> > > timestamp,
> > > > > or is
> > > > > > > > this
> > > > > > > > > > an
> > > > > > > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > > > >>> Bill
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J.
> > Sax <
> > > > > > > > > > > mjsax@apache.org>
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> I think it's not useful to record the
> > avg/mean;
> > > it
> > > > > > > > sensitive
> > > > > > > > > > to
> > > > > > > > > > > > > > >>>> outliers. We should rather track the median
> > > (50th
> > > > > > > > > percentile).
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Not sure if tracking min is useful, but I am
> > > also
> > > > > ok to
> > > > > > > > > track
> > > > > > > > > > > it.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> However, I find it odd to track 75th
> > percentile.
> > > > > > > Standard
> > > > > > > > > > > measures
> > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need
> > both,
> > > so
> > > > > maybe
> > > > > > > > > > picking
> > > > > > > > > > > > 90th
> > > > > > > > > > > > > > >>>> might be more useful?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> About the name: "staleness" wound really
> odd,
> > > and if
> > > > > > > fact
> > > > > > > > > the
> > > > > > > > > > > > metric
> > > > > > > > > > > > > > >>>> does capture "latency" so we should call it
> > > > > "latency". I
> > > > > > > > > > > > understand
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>> issue that we already have a latency metric.
> > So
> > > > > maybe we
> > > > > > > > > could
> > > > > > > > > > > > call it
> > > > > > > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> While I agree that we should include
> > > out-or-order
> > > > > data
> > > > > > > > (the
> > > > > > > > > > KIP
> > > > > > > > > > > > should
> > > > > > > > > > > > > > >>>> talk about `out-of-order` data, not `late`
> > data;
> > > > > data is
> > > > > > > > > only
> > > > > > > > > > > > `late`
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > >>>> it's out-of-order and if it's dropped), I
> > don't
> > > > > really
> > > > > > > > > > > understand
> > > > > > > > > > > > why
> > > > > > > > > > > > > > >>>> the new metric would help to configure grace
> > > period
> > > > > or
> > > > > > > > > > retention
> > > > > > > > > > > > time?
> > > > > > > > > > > > > > >>>> As you mention in the KIP, both are define
> as
> > > max
> > > > > > > > difference
> > > > > > > > > > of
> > > > > > > > > > > > > > >>>> `event-time - stream-time` and thus the new
> > > metric
> > > > > that
> > > > > > > > > takes
> > > > > > > > > > > > > > >>>> system-/wallclock-time into account does not
> > > seem to
> > > > > > > help
> > > > > > > > at
> > > > > > > > > > > all.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to
> > > Measure
> > > > > > > > > Latency"
> > > > > > > > > > by
> > > > > > > > > > > > Gil
> > > > > > > > > > > > > > >>>> Tene:
> > > https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> -Matthias
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> It seems like there would still be plenty
> of
> > > use
> > > > > cases
> > > > > > > > for
> > > > > > > > > > > > recording
> > > > > > > > > > > > > > >>>>> this metric at all processors and not just
> > > stateful
> > > > > > > ones,
> > > > > > > > > but
> > > > > > > > > > > I'm
> > > > > > > > > > > > > > >> happy
> > > > > > > > > > > > > > >>>>> to suspend my arguments for now. Since
> you're
> > > > > proposing
> > > > > > > > to
> > > > > > > > > > keep
> > > > > > > > > > > > > > >>>>> them at the processor-node level, it will
> be
> > > > > seamless
> > > > > > > > later
> > > > > > > > > > to
> > > > > > > > > > > > add
> > > > > > > > > > > > > > >>>>> in the stateless processors if we want. As
> a
> > > wise
> > > > > man
> > > > > > > > once
> > > > > > > > > > > said,
> > > > > > > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Regarding the time measurement, it's an
> > > > > implementation
> > > > > > > > > detail
> > > > > > > > > > > > > > >>>>> we don't need to consider in the KIP.
> > > > > Nevertheless, I'd
> > > > > > > > > > greatly
> > > > > > > > > > > > > > >>>>> prefer to measure the system time again
> when
> > > > > recording
> > > > > > > > the
> > > > > > > > > > > > > > >>>>> metric. I don't think we've seen any
> evidence
> > > that
> > > > > > > proves
> > > > > > > > > > this
> > > > > > > > > > > > > > >>>>> would harm performance, and the amount of
> > > > > inaccuracy
> > > > > > > > using
> > > > > > > > > > > > > > >>>>> the cached system time could incur is
> > actually
> > > > > > > > substantial.
> > > > > > > > > > > But,
> > > > > > > > > > > > > > >>>>> if you want to just "not mention this" in
> the
> > > KIP,
> > > > > we
> > > > > > > can
> > > > > > > > > > defer
> > > > > > > > > > > > to
> > > > > > > > > > > > > > >>>>> the actual PR discussion, at which time
> we're
> > > in a
> > > > > > > better
> > > > > > > > > > > > position
> > > > > > > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Along the lines of the measurement accuracy
> > > > > discussion,
> > > > > > > > one
> > > > > > > > > > > > > > >>>>> minor thought I had is that maybe we should
> > > > > consider
> > > > > > > > > > measuring
> > > > > > > > > > > > > > >>>>> the task staleness metric at the sink,
> rather
> > > than
> > > > > the
> > > > > > > > > > source,
> > > > > > > > > > > so
> > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > >>>>> it includes the processing latency of the
> > task
> > > > > itself,
> > > > > > > > not
> > > > > > > > > > just
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>> latency
> > > > > > > > > > > > > > >>>>> of everything up to, but not including, the
> > > task
> > > > > (which
> > > > > > > > > seems
> > > > > > > > > > > > > > >> confusing
> > > > > > > > > > > > > > >>>>> for users). I guess this could also be an
> > > > > > > implementation
> > > > > > > > > > > detail,
> > > > > > > > > > > > > > >>> though.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > > > > > > >>>>> -John
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie
> > > Blee-Goldman
> > > > > > > > wrote:
> > > > > > > > > > > > > > >>>>>> Hey all,
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> After discussing with Bruno I'd like to
> > > propose a
> > > > > > > small
> > > > > > > > > > > > amendment,
> > > > > > > > > > > > > > >>>>>> which is to record the
> processor-node-level
> > > > > metrics
> > > > > > > only
> > > > > > > > > for
> > > > > > > > > > > > > > >>> *stateful*
> > > > > > > > > > > > > > >>>>>> *operators*. They would still be
> considered
> > a
> > > > > > > > > > > > "processor-node-level"
> > > > > > > > > > > > > > >>>>>> metric and not a "state-store-level"
> metric
> > > as the
> > > > > > > > > staleness
> > > > > > > > > > > is
> > > > > > > > > > > > > > >> still
> > > > > > > > > > > > > > >>>>>> a property of the node rather than of the
> > > state
> > > > > > > itself.
> > > > > > > > > > > > However, it
> > > > > > > > > > > > > > >>>> seems
> > > > > > > > > > > > > > >>>>>> that this information is primarily useful
> > for
> > > > > stateful
> > > > > > > > > > > operators
> > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > >>>> might
> > > > > > > > > > > > > > >>>>>> be exposing state via IQ or otherwise
> > > dependent
> > > > > on the
> > > > > > > > > > record
> > > > > > > > > > > > time
> > > > > > > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> It's worth calling out that recent
> > performance
> > > > > > > > > improvements
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>> metrics
> > > > > > > > > > > > > > >>>>>> framework mean that we no longer fetch the
> > > system
> > > > > time
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > > > > > >> operator
> > > > > > > > > > > > > > >>>>>> level, but only once per task. In other
> > words
> > > the
> > > > > > > system
> > > > > > > > > > time
> > > > > > > > > > > > is not
> > > > > > > > > > > > > > >>>> updated
> > > > > > > > > > > > > > >>>>>> between each process as a record flows
> > > through the
> > > > > > > > > > > subtopology,
> > > > > > > > > > > > so
> > > > > > > > > > > > > > >>>>>> debugging the processor-level latency via
> > the
> > > > > > > > stateleness
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > >> be
> > > > > > > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > > > > > > operator-level
> > > > > > > > > > > metrics
> > > > > > > > > > > > are
> > > > > > > > > > > > > > >>> not
> > > > > > > > > > > > > > >>>>>> *useful* relative to the task-level
> metric.
> > > > > Upstream
> > > > > > > > > caching
> > > > > > > > > > > > and/or
> > > > > > > > > > > > > > >>>>>> suppression
> > > > > > > > > > > > > > >>>>>> can still cause a record's staleness at
> some
> > > > > > > downstream
> > > > > > > > > > > stateful
> > > > > > > > > > > > > > >>>> operator
> > > > > > > > > > > > > > >>>>>> to deviate from the task-level staleness
> > > > > (recorded at
> > > > > > > > the
> > > > > > > > > > > source
> > > > > > > > > > > > > > >>> node).
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> Please let me know if you have any
> concerns
> > > about
> > > > > this
> > > > > > > > > > change.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno
> > Cadonna
> > > <
> > > > > > > > > > > > bruno@confluent.io>
> > > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > > > > > > >>>>>>> I think we do not need it now. If we need
> > > it, we
> > > > > can
> > > > > > > > add
> > > > > > > > > > it.
> > > > > > > > > > > > Here
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >>>>>>> old truism applies: Adding is always
> easier
> > > than
> > > > > > > > > removing.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > > > > > > >>>>>>> I think it is good to have the staleness
> > > metrics
> > > > > also
> > > > > > > > on
> > > > > > > > > > > > > > >>>>>>> processor-node-level. If we do not want
> to
> > > record
> > > > > > > them
> > > > > > > > on
> > > > > > > > > > all
> > > > > > > > > > > > > > >>>>>>> processor nodes, you could restrict the
> > > > > recording to
> > > > > > > > > > stateful
> > > > > > > > > > > > > > >>>>>>> processor-nodes, since those are the ones
> > > that
> > > > > would
> > > > > > > > > > benefit
> > > > > > > > > > > > most
> > > > > > > > > > > > > > >>> from
> > > > > > > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> Best,
> > > > > > > > > > > > > > >>>>>>> Bruno
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie
> > > > > Blee-Goldman <
> > > > > > > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to
> > align
> > > > > with the
> > > > > > > > > > current
> > > > > > > > > > > > > > >>> metrics.
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> Is it better to conform than to be
> right?
> > > > > History
> > > > > > > has
> > > > > > > > a
> > > > > > > > > > lot
> > > > > > > > > > > > to say
> > > > > > > > > > > > > > >>> on
> > > > > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > > > > >>>>>>>> matter
> > > > > > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies
> to
> > > the
> > > > > fine
> > > > > > > > > > details
> > > > > > > > > > > of
> > > > > > > > > > > > > > >>> metrics
> > > > > > > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> More seriously, I figured if people are
> > > looking
> > > > > at
> > > > > > > > this
> > > > > > > > > > > metric
> > > > > > > > > > > > > > >>> they're
> > > > > > > > > > > > > > >>>>>>>> likely to
> > > > > > > > > > > > > > >>>>>>>> be looking at all the others. Then
> naming
> > > this
> > > > > one
> > > > > > > > > "-mean"
> > > > > > > > > > > > would
> > > > > > > > > > > > > > >>>> probably
> > > > > > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg"
> > > suffix in
> > > > > the
> > > > > > > > > other
> > > > > > > > > > > > metrics
> > > > > > > > > > > > > > >>> has
> > > > > > > > > > > > > > >>>> a
> > > > > > > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> As for the percentiles, I actually like
> > p99
> > > (and
> > > > > > > p75)
> > > > > > > > > > > better.
> > > > > > > > > > > > I'll
> > > > > > > > > > > > > > >>>> swap
> > > > > > > > > > > > > > >>>>>>>> that out
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John
> > > Roesler <
> > > > > > > > > > > > vvcephei@apache.org
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is
> > > there a
> > > > > > > > reason
> > > > > > > > > to
> > > > > > > > > > > > choose
> > > > > > > > > > > > > > >>>> "avg"
> > > > > > > > > > > > > > >>>>>>>>> instead
> > > > > > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid,
> > and
> > > I
> > > > > might
> > > > > > > be
> > > > > > > > > > > > > > >> oversensitive
> > > > > > > > > > > > > > >>>>>>> because
> > > > > > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but
> > it
> > > > > strikes
> > > > > > > me
> > > > > > > > > > that
> > > > > > > > > > > > "avg"
> > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > >>>>>>>>> actually
> > > > > > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of
> > > > > statistics,
> > > > > > > > > > whereas
> > > > > > > > > > > > "mean"
> > > > > > > > > > > > > > >>> is
> > > > > > > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg",
> but
> > > none
> > > > > with
> > > > > > > > > > "mean";
> > > > > > > > > > > > was
> > > > > > > > > > > > > > >>> that
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Regarding the names of the percentile,
> I
> > > > > actually
> > > > > > > > > > couldn't
> > > > > > > > > > > > find
> > > > > > > > > > > > > > >>> _any_
> > > > > > > > > > > > > > >>>>>>> other
> > > > > > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there
> a
> > > > > reason to
> > > > > > > > > choose
> > > > > > > > > > > > "99th"
> > > > > > > > > > > > > > >> as
> > > > > > > > > > > > > > >>>>>>> opposed
> > > > > > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is
> > not a
> > > > > > > > criticism,
> > > > > > > > > > I'm
> > > > > > > > > > > > just
> > > > > > > > > > > > > > >>>>>>> primarily
> > > > > > > > > > > > > > >>>>>>>>> asking
> > > > > > > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > > > > > >>>>>>>>> -John
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie
> > > > > Blee-Goldman
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the
> min
> > > > > metric
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > sake of
> > > > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the
> > mean
> > > (or
> > > > > 50th
> > > > > > > > > > > > percentile)
> > > > > > > > > > > > > > >>> but I
> > > > > > > > > > > > > > >>>>>>> can
> > > > > > > > > > > > > > >>>>>>>>> see
> > > > > > > > > > > > > > >>>>>>>>>> how users might expect it and find it
> a
> > > bit
> > > > > > > > > disorienting
> > > > > > > > > > > > not to
> > > > > > > > > > > > > > >>>>>>> have. So
> > > > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > > > > > > percentile)*
> > > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > > > > > > percentile)*
> > > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms]
> *(mean)*
> > > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John
> > > Roesler <
> > > > > > > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity
> to
> > > > > embarrass
> > > > > > > > > > myself.
> > > > > > > > > > > I
> > > > > > > > > > > > > > >> guess
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>> mean
> > > > > > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the
> > > > > median, but
> > > > > > > > > > neither
> > > > > > > > > > > > are
> > > > > > > > > > > > > > >> as
> > > > > > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles
> > > (99th
> > > > > and
> > > > > > > > max).
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any
> > SLAs,
> > > but
> > > > > it
> > > > > > > > does
> > > > > > > > > > > round
> > > > > > > > > > > > out
> > > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > > >>>>>>>>> mental
> > > > > > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve
> > always
> > > > > graphed
> > > > > > > > min
> > > > > > > > > > > along
> > > > > > > > > > > > with
> > > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > > > > >>>>>>>>>>> metrics to help me understand how
> fast
> > > the
> > > > > system
> > > > > > > > can
> > > > > > > > > > be,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > >>>>>>> helps
> > > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a
> > > > > relatively
> > > > > > > > > > > inexpensive
> > > > > > > > > > > > > > >> metric
> > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just
> > > throw
> > > > > it in.
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18,
> Sophie
> > > > > > > Blee-Goldman
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > > > > > > >>>>>>>>>>>> I was considering it as the
> > "end-to-end
> > > > > latency
> > > > > > > > *up*
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if
> > > that
> > > > > drives
> > > > > > > > the
> > > > > > > > > > > point
> > > > > > > > > > > > > > >> home
> > > > > > > > > > > > > > >>>>>>>>> better.
> > > > > > > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is
> > > > > received by
> > > > > > > > that
> > > > > > > > > > > > task" --
> > > > > > > > > > > > > > >>>>>>> will
> > > > > > > > > > > > > > >>>>>>>>> update
> > > > > > > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case
> where
> > > the
> > > > > min
> > > > > > > > would
> > > > > > > > > > > > actually
> > > > > > > > > > > > > > >> be
> > > > > > > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I
> > don't
> > > > > feel
> > > > > > > > > strongly
> > > > > > > > > > > > that we
> > > > > > > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the
> > start.
> > > Can
> > > > > you
> > > > > > > > > > > enlighten
> > > > > > > > > > > > me
> > > > > > > > > > > > > > >>>>>>> with an
> > > > > > > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about
> the
> > > > > overhead
> > > > > > > of
> > > > > > > > > > > adding
> > > > > > > > > > > > > > >>>>>>> multiple
> > > > > > > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to
> > indicate
> > > > > what
> > > > > > > kind
> > > > > > > > > of
> > > > > > > > > > > > > > >>> performance
> > > > > > > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the
> > > 50th
> > > > > > > > percentile
> > > > > > > > > > > > would be
> > > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > > >>>>>>>>> median
> > > > > > > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the
> > mean
> > > > > *and*
> > > > > > > the
> > > > > > > > > 50th
> > > > > > > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John
> > > > > Roesler <
> > > > > > > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP,
> > > Sophie,
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Just some comments on the
> discussion
> > so
> > > > > far:
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter
> > > whether
> > > > > we
> > > > > > > > report
> > > > > > > > > > > > "spans"
> > > > > > > > > > > > > > >> or
> > > > > > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in
> > practice,
> > > > > some of
> > > > > > > > the
> > > > > > > > > > > spans
> > > > > > > > > > > > are
> > > > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like
> > time
> > > > > spent
> > > > > > > > > waiting
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > > > > > > >>>>>>>>>>>>> time from the event happening to
> the
> > > ETL
> > > > > > > producer
> > > > > > > > > > > > choosing to
> > > > > > > > > > > > > > >>>>>>> send
> > > > > > > > > > > > > > >>>>>>>>> it,
> > > > > > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive
> > buffers,
> > > > > etc.,
> > > > > > > etc.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically
> > > easier to
> > > > > > > > compute
> > > > > > > > > > > spans
> > > > > > > > > > > > by
> > > > > > > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute
> > e2e
> > > > > > > latencies
> > > > > > > > > by
> > > > > > > > > > > > adding
> > > > > > > > > > > > > > >>>>>>> spans.
> > > > > > > > > > > > > > >>>>>>>>> You
> > > > > > > > > > > > > > >>>>>>>>>>>>> can even just consider that the
> span
> > > > > > > computation
> > > > > > > > > from
> > > > > > > > > > > e2e
> > > > > > > > > > > > > > >>>>>>> always
> > > > > > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers,
> > > whereas
> > > > > > > > computing
> > > > > > > > > > e2e
> > > > > > > > > > > > > > >> latency
> > > > > > > > > > > > > > >>>>>>>>> from
> > > > > > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the
> spans
> > > > > leading
> > > > > > > up
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > end
> > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > >>>>>>>>> care
> > > > > > > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer
> to
> > > have
> > > > > > > spans
> > > > > > > > > when
> > > > > > > > > > > > they
> > > > > > > > > > > > > > >> are
> > > > > > > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e
> latency
> > > is a
> > > > > more
> > > > > > > > > > general
> > > > > > > > > > > > > > >>>>>>> measurement
> > > > > > > > > > > > > > >>>>>>>>>>>>> that basically every
> > person/application
> > > > > cares
> > > > > > > > about
> > > > > > > > > > and
> > > > > > > > > > > > > > >> should
> > > > > > > > > > > > > > >>>>>>> be
> > > > > > > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to
> provide
> > > more
> > > > > > > value
> > > > > > > > to
> > > > > > > > > > > more
> > > > > > > > > > > > > > >>>>>>> people if
> > > > > > > > > > > > > > >>>>>>>>> we
> > > > > > > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding
> > > > > "record-staleness" as
> > > > > > > a
> > > > > > > > > > name,
> > > > > > > > > > > I
> > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > >>>>>>> I
> > > > > > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other
> > peoples'
> > > > > > > > intuition.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric
> is
> > > nice,
> > > > > > > since
> > > > > > > > > the
> > > > > > > > > > > > inside
> > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > > >>>>>>>>> task
> > > > > > > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of
> > > latency
> > > > > in
> > > > > > > some
> > > > > > > > > > > cases.
> > > > > > > > > > > > > > >> Plus,
> > > > > > > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really
> > > wanted to
> > > > > > > know
> > > > > > > > > (for
> > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>> purposes
> > > > > > > > > > > > > > >>>>>>>>> of
> > > > > > > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes
> > source
> > > > > events
> > > > > > > to
> > > > > > > > > > "show
> > > > > > > > > > > > up" at
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at
> > every
> > > > > > > processor
> > > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics
> at
> > > the
> > > > > node
> > > > > > > > > level.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to
> > > record
> > > > > also.
> > > > > > > > > Even
> > > > > > > > > > if
> > > > > > > > > > > > it's
> > > > > > > > > > > > > > >> a
> > > > > > > > > > > > > > >>>>>>> poor
> > > > > > > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of
> > > people
> > > > > might
> > > > > > > > > > expect
> > > > > > > > > > > > to see
> > > > > > > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min".
> Is
> > > there
> > > > > a
> > > > > > > > reason
> > > > > > > > > > to
> > > > > > > > > > > > leave
> > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > >>>>>>>>> off?
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > > > > > > >>>>>>>>>>>>> I agree we should include late
> > records
> > > > > (though
> > > > > > > > not
> > > > > > > > > > the
> > > > > > > > > > > > ones
> > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when
> there
> > > are
> > > > > > > > > legitimately
> > > > > > > > > > > > some
> > > > > > > > > > > > > > >>>>>>> records
> > > > > > > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is
> the
> > > whole
> > > > > > > point
> > > > > > > > > of
> > > > > > > > > > > > these
> > > > > > > > > > > > > > >>>>>>> metrics.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any
> > > other
> > > > > > > > feedback,
> > > > > > > > > > > other
> > > > > > > > > > > > > > >> than
> > > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58,
> > > Guozhang
> > > > > Wang
> > > > > > > > > wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few
> > quick
> > > > > > > thoughts:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes
> > > both
> > > > > the
> > > > > > > > > > processing
> > > > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in
> > > > > intermediate
> > > > > > > > > > topics.
> > > > > > > > > > > I
> > > > > > > > > > > > have
> > > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above
> > > that the
> > > > > > > > latency
> > > > > > > > > > > > metric of
> > > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > > >>>>>>>>> task A
> > > > > > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of
> the
> > > > > > > > sub-topology
> > > > > > > > > > > up-to
> > > > > > > > > > > > but
> > > > > > > > > > > > > > >>>>>>> not
> > > > > > > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a
> bit
> > > weird.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from
> the
> > > name
> > > > > > > > "latency"
> > > > > > > > > > > > itself,
> > > > > > > > > > > > > > >>>>>>> since
> > > > > > > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > > > > > > >>>>>>>>>>>>>> already have several "latency"
> > metrics
> > > > > already
> > > > > > > > > which
> > > > > > > > > > > are
> > > > > > > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a
> record
> > /
> > > etc,
> > > > > > > while
> > > > > > > > > > here
> > > > > > > > > > > > we are
> > > > > > > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the
> > record
> > > > > > > timestamp.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it
> as
> > > > > > > > > > > "record-staleness"
> > > > > > > > > > > > > > >>>>>>> (note we
> > > > > > > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric),
> in
> > > which
> > > > > > > case
> > > > > > > > > > > > recording at
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>>>>> system-time before we start
> > > processing the
> > > > > > > > record
> > > > > > > > > > > sounds
> > > > > > > > > > > > > > >> more
> > > > > > > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm
> wondering
> > > if the
> > > > > > > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that
> > we
> > > > > already
> > > > > > > > > have a
> > > > > > > > > > > > > > >>>>>>> task-level
> > > > > > > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a
> > specific
> > > > > node's
> > > > > > > e2e
> > > > > > > > > > > > latency is
> > > > > > > > > > > > > > >>>>>>>>> similar
> > > > > > > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency +
> task-level
> > > > > processing
> > > > > > > > > > > latency.
> > > > > > > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > > > > > > >>>>>>>>>>>>>> having a task-level
> record-staleness
> > > > > metric is
> > > > > > > > > > > > sufficient.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM
> > > Sophie
> > > > > > > > > > Blee-Goldman <
> > > > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a
> > > particularly
> > > > > > > > useful
> > > > > > > > > > > gauge
> > > > > > > > > > > > for
> > > > > > > > > > > > > > >>>>>>> this
> > > > > > > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at
> > > putting an
> > > > > > > > *upper
> > > > > > > > > > > > *bound on
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose
> > > > > percentiles
> > > > > > > > > that
> > > > > > > > > > > > would
> > > > > > > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will
> > > > > experience,
> > > > > > > > and
> > > > > > > > > > > what
> > > > > > > > > > > > > > >>>>>>> *close
> > > > > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these
> > > specific
> > > > > > > > numbers
> > > > > > > > > > and
> > > > > > > > > > > > > > >>>>>>> could be
> > > > > > > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in
> > > hearing
> > > > > > > from
> > > > > > > > > > users
> > > > > > > > > > > on
> > > > > > > > > > > > > > >>>>>>> this.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include
> the
> > > > > > > "hop-to-hop
> > > > > > > > > > > > latency" in
> > > > > > > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves
> by
> > > > > > > subtracting
> > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>> previous
> > > > > > > > > > > > > > >>>>>>>>> node's
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we
> > could
> > > do
> > > > > it
> > > > > > > > either
> > > > > > > > > > way
> > > > > > > > > > > > since
> > > > > > > > > > > > > > >>>>>>>>> you can
> > > > > > > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I
> > > think
> > > > > the
> > > > > > > > > > > end-to-end
> > > > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation
> is
> > > not
> > > > > to
> > > > > > > > debug
> > > > > > > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > > > > > > >>>>>>>>> in the
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long
> > it
> > > > > takes
> > > > > > > > > arecord
> > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this
> > > might
> > > > > be
> > > > > > > > useful
> > > > > > > > > > for
> > > > > > > > > > > > users
> > > > > > > > > > > > > > >>>>>>>>> who are
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was
> just
> > > > > produced
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > >>>>>>> included
> > > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side
> > effect
> > > --
> > > > > but
> > > > > > > > > maybe I
> > > > > > > > > > > > didn't
> > > > > > > > > > > > > > >>>>>>> make
> > > > > > > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should
> address
> > > this
> > > > > in
> > > > > > > the
> > > > > > > > > KIP.
> > > > > > > > > > > The
> > > > > > > > > > > > > > >>>>>>> short
> > > > > > > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I
> > > added a
> > > > > > > > paragraph
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>> end
> > > > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the
> > > reasoning
> > > > > > > here,
> > > > > > > > > > please
> > > > > > > > > > > > let
> > > > > > > > > > > > > > >>>>>>> me
> > > > > > > > > > > > > > >>>>>>>>> know
> > > > > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to
> the
> > > > > existing
> > > > > > > > > metric
> > > > > > > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > > > > > > Node#process
> > > > > > > > > > method
> > > > > > > > > > > > to
> > > > > > > > > > > > > > >>>>>>> run
> > > > > > > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative
> > to
> > > the
> > > > > > > event
> > > > > > > > > > > > timestamp.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision
> > > there is
> > > > > > > pretty
> > > > > > > > > > > > confusing
> > > > > > > > > > > > > > >>>>>>> so
> > > > > > > > > > > > > > >>>>>>>>> I've
> > > > > > > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to
> > > > > "end-to-end-latency"
> > > > > > > > > which I
> > > > > > > > > > > > feel
> > > > > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM
> > > Boyang
> > > > > Chen
> > > > > > > <
> > > > > > > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie.
> Getting
> > > the
> > > > > E2E
> > > > > > > > > latency
> > > > > > > > > > > is
> > > > > > > > > > > > > > >>>>>>>>> important
> > > > > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of
> > the
> > > > > > > > application.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the
> rational
> > of
> > > > > picking
> > > > > > > > 75,
> > > > > > > > > > 99
> > > > > > > > > > > > and
> > > > > > > > > > > > > > >>>>>>> max
> > > > > > > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we
> use
> > > 50, 90
> > > > > > > > > > percentile
> > > > > > > > > > > as
> > > > > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being
> > > computed is
> > > > > > > > > > cumulative,
> > > > > > > > > > > > I.E
> > > > > > > > > > > > > > >>>>>>> if a
> > > > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) =
> > > > > T(B->C) +
> > > > > > > > P(B)
> > > > > > > > > =
> > > > > > > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents
> > the
> > > > > captured
> > > > > > > > > > > latency,
> > > > > > > > > > > > > > >>>>>>> and T()
> > > > > > > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the
> > records
> > > > > between
> > > > > > > > two
> > > > > > > > > > > nodes,
> > > > > > > > > > > > > > >>>>>>>>> including
> > > > > > > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose,
> > maybe
> > > > > having
> > > > > > > > > T(B->C)
> > > > > > > > > > > and
> > > > > > > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop
> > > latency",
> > > > > > > > > otherwise
> > > > > > > > > > > if
> > > > > > > > > > > > > > >>>>>>> there
> > > > > > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are
> > > > > affected in
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > > >>>>>>> time.
> > > > > > > > > > > > > > >>>>>>>>> In the
> > > > > > > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is
> > > meaningful
> > > > > only
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>> record
> > > > > > > > > > > > > > >>>>>>>>> exits
> > > > > > > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole
> time
> > > this
> > > > > > > record
> > > > > > > > > > spent
> > > > > > > > > > > > > > >>>>>>> inside
> > > > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate
> > > > > treatment
> > > > > > > for
> > > > > > > > > > sink
> > > > > > > > > > > > > > >>>>>>> nodes and
> > > > > > > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only
> > > count
> > > > > the
> > > > > > > time
> > > > > > > > > > > > receiving
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a
> > > solution
> > > > > here,
> > > > > > > > > just
> > > > > > > > > > > > want to
> > > > > > > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is
> > > reasonable.
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor
> late
> > > > > arrival
> > > > > > > > > records
> > > > > > > > > > > as
> > > > > > > > > > > > > > >>>>>>> well,
> > > > > > > > > > > > > > >>>>>>>>> they
> > > > > > > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs
> > > when the
> > > > > > > > > > > out-of-order
> > > > > > > > > > > > > > >>>>>>>>> records are
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time
> records.
> > > > > Should we
> > > > > > > > > also
> > > > > > > > > > > > supply
> > > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user
> > > should
> > > > > just
> > > > > > > > take
> > > > > > > > > > > care
> > > > > > > > > > > > of
> > > > > > > > > > > > > > >>>>>>> it
> > > > > > > > > > > > > > >>>>>>>>> by
> > > > > > > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics,
> we
> > > > > haven't
> > > > > > > > > > discussed
> > > > > > > > > > > > its
> > > > > > > > > > > > > > >>>>>>>>> relation
> > > > > > > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency
> > metrics,
> > > > > could
> > > > > > > you
> > > > > > > > > add
> > > > > > > > > > > > some
> > > > > > > > > > > > > > >>>>>>>>> context
> > > > > > > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to
> > use
> > > > > which`
> > > > > > > > > > tutorial
> > > > > > > > > > > > for
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM
> > > Sophie
> > > > > > > > > > Blee-Goldman
> > > > > > > > > > > <
> > > > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion
> > on
> > > > > KIP-613
> > > > > > > > > which
> > > > > > > > > > > aims
> > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > >>>>>>>>> add
> > > > > > > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams.
> > Please
> > > > > take a
> > > > > > > > look:
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > -- Guozhang
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
Thanks Sophie, I'm +1 on the updated KIP.

On Mon, May 18, 2020 at 8:32 AM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> @Bruno
> No, this would be after inserting in the cache/state store/etc -- basically
> once
> all operations have completed such that the record would be present if
> queried after that time.
>
> @Guozhang
> I'm glad you suggested the TRACE level. It seems we've been
> heading in that direction for a while so I'm happy to add it as part of
> this KIP. Especially if it unblocks the question of whether to update the
> system time (for now).
>
> On Mon, May 18, 2020 at 1:09 AM Bruno Cadonna <br...@confluent.io> wrote:
>
> > Hi Sophie,
> >
> > Just a question for my understanding. When you say " t_A be the time
> > when it is finished being processed by the aggregator node", this
> > means before caching (or putting into the state store if caching is
> > disabled), right?
> >
> > Best,
> > Bruno
> >
> > On Sat, May 16, 2020 at 6:57 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> > >
> > > We do have a big JIRA for making sensor recording to be sufficient. But
> > it
> > > is a bit hard to try to reduce the num.system clock calls in general
> (but
> > > admittedly, in nowadays OS they are supposed to be very cheap as well),
> > > I've just recently merged a small PR trying to make less
> > > SystemTime.currentMilliseconds calls whenever possible, but it is hard
> to
> > > enforce that for all callers.
> > >
> > > Guozhang
> > >
> > > On Sat, May 16, 2020 at 9:46 AM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > FWIW, I’d be fine with TRACE, if it makes the processor-level metrics
> > more
> > > > palatable to everyone.
> > > >
> > > > Ultimately, I think this is just kicking the can down the road, and
> we
> > > > should really just refactor our sensors/metrics to be lower overhead.
> > But
> > > > that’s too much work to consider right before the release so for now,
> > we
> > > > can kick that can just a little farther.
> > > >
> > > > Thanks for proposing a solution, Guozhang!
> > > >
> > > > -John
> > > >
> > > > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> > > > > Cool, that's good points -- we should not let our current metrics
> > > > proposal
> > > > > to depend on future roadmaps, since they can still be removed later
> > if
> > > > ever
> > > > > becoming not valuable after all.
> > > > >
> > > > > On the other hand, what do you think about introducing TRACE level
> > > > metrics
> > > > > recording and make the the processor-node level metrics on TRACE?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <
> > sophie@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Yeah, I had proposed to replace the "source + sink" task-level
> > metric
> > > > > > with "sink only" task-level metrics plus the stateful node-level
> > ones.
> > > > > >
> > > > > > I completely agree that this proposal should and would look
> > different
> > > > > > once we complete those two proposals, ie decouple caching and
> > > > > > consolidate suppression. But as it is right now, caching and
> > > > suppression
> > > > > > cause the actual time-of-processing of records to vary greatly
> > > > throughout
> > > > > > a subtopology such that the delta between a source node and an
> > > > intermediate
> > > > > > one is potentially quite large.
> > > > > >
> > > > > > For this exact reason I would argue that this metric is still
> > valuable
> > > > at
> > > > > > the node-level
> > > > > >  even if the "current time" is actually only updated once for a
> > task.
> > > > > > Obviously this
> > > > > > would not convey any information about the actual processing
> > latency,
> > > > but
> > > > > > as you
> > > > > > stated, this is likely to be a small delta on the scale of
> > > > milliseconds.
> > > > > > But the latency
> > > > > > due to a suppression can be on the order of minutes, hours, or
> even
> > > > days!
> > > > > > If
> > > > > > there is more than one suppression or caching in a subtopology,
> the
> > > > source
> > > > > > and sink
> > > > > > e2e latency could be wildly misrepresentative of the e2e latency
> > of a
> > > > node
> > > > > > in the middle.
> > > > > >
> > > > > > Anyways, I removed this last point from the current KIP document
> > > > because I
> > > > > > felt
> > > > > > it was ultimately an implementation detail. But the larger point
> I
> > > > would
> > > > > > argue is that:
> > > > > > if we are truly worried about the performance hit of fetching the
> > > > system
> > > > > > time at every
> > > > > > node, we should just not fetch the system time at every node
> > instead of
> > > > > > dropping the
> > > > > > metric altogether. It may be slightly off but only on the order
> of
> > ms
> > > > and
> > > > > > not on the order
> > > > > > of hours, and it's still more useful than no metric at all
> > > > > >
> > > > > >
> > > > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <
> wangguoz@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Sophie,
> > > > > > >
> > > > > > > I think your motivation makes sense, in fact before KIP-444 we
> > used
> > > > to
> > > > > > have
> > > > > > > processor-node-level process latency as DEBUG which would
> > > > holistically
> > > > > > > address the motivation but we removed it since it is indeed
> quite
> > > > > > expensive
> > > > > > > (I do not have concrete numbers at hand, but basically you'd
> > have to
> > > > call
> > > > > > > system-wall-clock time a couple of time for each record, each
> > > > processor
> > > > > > > node), and in the long run I feel such scenarios would be less
> > > > common if
> > > > > > we
> > > > > > > 1) decouple caching from emitting and 2) always put suppression
> > right
> > > > > > > before sink only, and that's also why I've proposed to remove
> > them in
> > > > > > > KIP-444.
> > > > > > >
> > > > > > > I missed an update in your wiki that right now you've updated
> the
> > > > > > > task-level metric to be recorded at the sink node, and thought
> > it is
> > > > > > still
> > > > > > > at the source node, but assuming it is still the case, then
> > > > practically
> > > > > > any
> > > > > > > state store's staleness of the task could roughly be measured
> as
> > that
> > > > > > > task-level staleness plus a small delta (usually tens of
> mills),
> > and
> > > > the
> > > > > > > down stream task B's staleness - up stream task A's staleness
> > would
> > > > > > capture
> > > > > > > the suppression effect at the end of A plus the intermediate
> > topic
> > > > > > > producing -> consuming latency, which usually would not require
> > to be
> > > > > > > distinguished.
> > > > > > >
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <
> > > > sophie@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Guozhang,
> > > > > > > >
> > > > > > > > Thanks for the response. I meant to ask this earlier and
> > forgot,
> > > > but do
> > > > > > > you
> > > > > > > > have any data or benchmarking results on hand for the
> > performance
> > > > hit
> > > > > > we
> > > > > > > > took due to the metrics? Or maybe better yet, on the
> > performance we
> > > > > > > gained
> > > > > > > > back by dropping some metrics? I think it would help us to
> > better
> > > > > > > > understand
> > > > > > > > the tradeoffs with some concrete numbers to compare.
> > > > > > > >
> > > > > > > > I guess I'm inclined to want to include these metrics at the
> > > > operator
> > > > > > > level
> > > > > > > > on
> > > > > > > > DEBUG since, in my mind, that's exactly what DEBUG is for:
> > > > performance-
> > > > > > > > focused users will presumably run at INFO level and even
> those
> > who
> > > > do
> > > > > > > > use DEBUG may only do so temporarily, when they actually have
> > > > something
> > > > > > > > to debug.
> > > > > > > >
> > > > > > > > Of course, I'm not running a Streams app myself so all this
> is
> > just
> > > > > > > > conjecture.
> > > > > > > > But it seems reasonable to interpret the two metrics levels
> as
> > > > "minimal
> > > > > > > > useful
> > > > > > > > information" and "all information that provides insight into
> > the
> > > > > > system"
> > > > > > > >
> > > > > > > > If we can agree there, the question becomes "does this
> provide
> > any
> > > > > > > > additional
> > > > > > > > insight?" I would still argue that it does. To give a
> concrete
> > > > example,
> > > > > > > > let's say
> > > > > > > > the user has a basic
> > > > > > > >
> > > > > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > > > > > >
> > > > > > > > subtopology, the application performing IQ on the
> transformer's
> > > > state.
> > > > > > > The
> > > > > > > > user
> > > > > > > > wants to know how long it's taking records to be reflected in
> > the
> > > > > > results
> > > > > > > > of a query.
> > > > > > > > With only task-level metrics, they can only guess based on
> the
> > e2e
> > > > > > > latency
> > > > > > > > of the
> > > > > > > > previous task and of this one. But neither of those is a good
> > > > > > > > representation of the
> > > > > > > > time it will take a record to be processed by the
> transformer:
> > the
> > > > > > > previous
> > > > > > > > task's
> > > > > > > > latency will not include the time for the record to go
> through
> > the
> > > > > > > > repartitioning,
> > > > > > > > and the current task's latency is measured after a
> > suppression. The
> > > > > > > > suppression
> > > > > > > > itself will introduce significant latency and doesn't provide
> > an
> > > > > > accurate
> > > > > > > > estimate.
> > > > > > > > Caching would also do what it does best, messing with our
> > > > understanding
> > > > > > > of
> > > > > > > > time :)
> > > > > > > >
> > > > > > > > It also means you can't measure the consumption latency, so
> > there
> > > > would
> > > > > > > be
> > > > > > > > no way to tell if the e2e latency is so high because the very
> > first
> > > > > > > > subtopology is
> > > > > > > > extremely slow, or the record was severely delayed in
> reaching
> > the
> > > > > > input
> > > > > > > > topic.
> > > > > > > > I guess my earlier proposal to include both source and sink
> > metrics
> > > > > > could
> > > > > > > > address this issue, but it seems to solve the problem more
> > > > holistically
> > > > > > > --
> > > > > > > > and for
> > > > > > > > more complex subtopologies -- by including metrics at the
> > processor
> > > > > > level
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey folks,
> > > > > > > > >
> > > > > > > > > I want to make a final fight on processor-node-level
> metrics
> > :)
> > > > Post
> > > > > > > > > KIP-444 we've actually removed a lot of node-level metrics
> > since
> > > > it
> > > > > > is
> > > > > > > > too
> > > > > > > > > expensive to record at that level and their values are not
> > proven
> > > > > > worth
> > > > > > > > it.
> > > > > > > > > Again I'd use the "we have to either enable all or none of
> > DEBUG
> > > > > > > metrics"
> > > > > > > > > card, e.g. if we want to look into per-task process-latency
> > > > which is
> > > > > > > > DEBUG,
> > > > > > > > > we'd have to record all process-node / state-store / cache
> > level
> > > > > > > metrics.
> > > > > > > > > On the other hand, within a task we usually do not have a
> > lot of
> > > > > > > stateful
> > > > > > > > > operator nodes and if we want to find if certain stateful
> > nodes
> > > > are
> > > > > > the
> > > > > > > > > bottleneck of process latency, that is usually discoverable
> > from
> > > > the
> > > > > > > > > state-store level read / write latencies already. So I'd
> > imagine
> > > > that
> > > > > > > > > suppose there's a sub-topology with multiple state stores
> in
> > it,
> > > > if
> > > > > > we
> > > > > > > > > found the task-level process latency is high, we can still
> > tell
> > > > which
> > > > > > > > state
> > > > > > > > > stores within it is the main bottleneck from the state
> store
> > > > read /
> > > > > > > write
> > > > > > > > > latency (I'm assuming the IO latency is always dominant
> > compared
> > > > with
> > > > > > > > > others).
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > > > > > sophie@confluent.io
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > @Matthias
> > > > > > > > > > > And that's what he says about the 95th percentile!
> > Imagine
> > > > what
> > > > > > he
> > > > > > > > > would
> > > > > > > > > > say about the 50th :P
> > > > > > > > > >
> > > > > > > > > > I should have kept watching another 10 minutes. He gets
> > around
> > > > to
> > > > > > > > > covering
> > > > > > > > > > the 50th, and let's
> > > > > > > > > > just say he is not a fan:
> > https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <
> > > > vvcephei@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks, Sophie!
> > > > > > > > > > >
> > > > > > > > > > > I think this makes perfect sense. It will be much more
> > > > intuitive
> > > > > > to
> > > > > > > > use
> > > > > > > > > > > the metric for the stated motivation this way.  I’d be
> in
> > > > favor
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > proposal after this update.
> > > > > > > > > > >
> > > > > > > > > > > Thanks again for taking this on,
> > > > > > > > > > > -John
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman
> > wrote:
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to clarify/modify one aspect of this KIP,
> > which
> > > > is to
> > > > > > > > record
> > > > > > > > > > the
> > > > > > > > > > > > staleness
> > > > > > > > > > > > at the *completion* of the record's processing by the
> > > > > > > operator/task
> > > > > > > > > in
> > > > > > > > > > > > question,
> > > > > > > > > > > > rather than on the intake. The task-level metrics
> will
> > be
> > > > > > > recorded
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > > > sink
> > > > > > > > > > > > node instead of at the source, and the operator-level
> > > > metrics
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > > recorded
> > > > > > > > > > > > at the end of the operation.
> > > > > > > > > > > >
> > > > > > > > > > > > The stated purpose and intended usefulness of this
> KIP
> > is
> > > > to
> > > > > > give
> > > > > > > > > > users a
> > > > > > > > > > > > way
> > > > > > > > > > > > to gauge roughly how long it takes for a record to be
> > > > reflected
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > "results",
> > > > > > > > > > > > whether these results are being read from an output
> > topic
> > > > or
> > > > > > > > through
> > > > > > > > > > IQ.
> > > > > > > > > > > To
> > > > > > > > > > > > take the IQ example, the results of a record are
> > obviously
> > > > not
> > > > > > > > > visible
> > > > > > > > > > > until
> > > > > > > > > > > > *after* that node has finished processing it. The
> > > > staleness,
> > > > > > > while
> > > > > > > > > > still
> > > > > > > > > > > > potentially
> > > > > > > > > > > > useful as it can impact the *way* a record is
> > processed in
> > > > a
> > > > > > > > stateful
> > > > > > > > > > and
> > > > > > > > > > > > time-
> > > > > > > > > > > > dependent operator, is not part of the problem this
> KIP
> > > > > > > > specifically
> > > > > > > > > > set
> > > > > > > > > > > out
> > > > > > > > > > > > to solve.
> > > > > > > > > > > >
> > > > > > > > > > > > In light of this I think it's appropriate to revert
> the
> > > > name
> > > > > > > change
> > > > > > > > > > back
> > > > > > > > > > > to
> > > > > > > > > > > > include
> > > > > > > > > > > > latency, since "staleness" as described above only
> > makes
> > > > sense
> > > > > > > when
> > > > > > > > > > > > measuring
> > > > > > > > > > > > relative to the arrival of a record at a task/node.
> I'd
> > > > propose
> > > > > > > to
> > > > > > > > > save
> > > > > > > > > > > the
> > > > > > > > > > > > term
> > > > > > > > > > > > "staleness" for that particular meaning, and adopt
> > > > Matthias's
> > > > > > > > > > suggestion
> > > > > > > > > > > of
> > > > > > > > > > > > "record-e2e-latency" for this.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for hanging in there all. Please let me know
> if
> > you
> > > > have
> > > > > > > any
> > > > > > > > > > > > concerns about
> > > > > > > > > > > > this change!
> > > > > > > > > > > >
> > > > > > > > > > > > Sophie
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > > > > > mjsax@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I am also happy with max/min/99/90. And I buy your
> > naming
> > > > > > > > argument
> > > > > > > > > > > about
> > > > > > > > > > > > > staleness vs latency.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Matthias
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > > > > > Hey Sophie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie
> > Blee-Goldman <
> > > > > > > > > > > > > sophie@confluent.io>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> @Matthias
> > > > > > > > > > > > > >> Regarding tracking the 50th percentile, I'll
> refer
> > > > you to
> > > > > > > the
> > > > > > > > > 4:53
> > > > > > > > > > > mark
> > > > > > > > > > > > > of
> > > > > > > > > > > > > >> the video
> > > > > > > > > > > > > >> *you* linked:
> https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> And that's what he says about the 95th
> percentile!
> > > > Imagine
> > > > > > > > what
> > > > > > > > > he
> > > > > > > > > > > would
> > > > > > > > > > > > > >> say about
> > > > > > > > > > > > > >> the 50th :P
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> But seriously, since we can't seem to agree that
> > the
> > > > mean
> > > > > > or
> > > > > > > > > 50th
> > > > > > > > > > > > > >> percentile is actually
> > > > > > > > > > > > > >> useful I'm inclined to resurrect my original
> > proposal,
> > > > > > > > neither.
> > > > > > > > > > But
> > > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > >> that's a good
> > > > > > > > > > > > > >> argument against the 75th, which I admittedly
> > chose
> > > > > > somewhat
> > > > > > > > > > > > > arbitrarily as
> > > > > > > > > > > > > >> an
> > > > > > > > > > > > > >> intermediate between the 50th and the higher
> > > > percentiles.
> > > > > > > How
> > > > > > > > > > about:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -max
> > > > > > > > > > > > > >> -p99
> > > > > > > > > > > > > >> -p90
> > > > > > > > > > > > > >> -min
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> with p50/mean still up for debate if anyone
> feels
> > > > strongly
> > > > > > > for
> > > > > > > > > > > either of
> > > > > > > > > > > > > >> them.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regarding the name, I've already flip-flopped on
> > this
> > > > so
> > > > > > I'm
> > > > > > > > > > > definitely
> > > > > > > > > > > > > >> still open to
> > > > > > > > > > > > > >> further arguments. But the reason for changing
> it
> > from
> > > > > > > > > > > > > end-to-end-latency
> > > > > > > > > > > > > >> (which
> > > > > > > > > > > > > >> is similar to what you propose) is that this
> > metric
> > > > > > > > technically
> > > > > > > > > > > reflects
> > > > > > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > > > > > >> the record is when it's *received* by the
> > operator,
> > > > not
> > > > > > when
> > > > > > > > > it's
> > > > > > > > > > > > > processed
> > > > > > > > > > > > > >> by the operator.
> > > > > > > > > > > > > >> It seemed like there was the potential for
> > confusion
> > > > that
> > > > > > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > > > > > >> represent the latency from the event creation to
> > the
> > > > time
> > > > > > > the
> > > > > > > > > > > processor
> > > > > > > > > > > > > is
> > > > > > > > > > > > > >> done
> > > > > > > > > > > > > >> processing it.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> @John
> > > > > > > > > > > > > >> I'd rather err on the side of "not-enough"
> > metrics as
> > > > we
> > > > > > can
> > > > > > > > > > always
> > > > > > > > > > > add
> > > > > > > > > > > > > >> this to the
> > > > > > > > > > > > > >> stateless metrics later on. If we decide to
> > measure
> > > > the
> > > > > > time
> > > > > > > > at
> > > > > > > > > > > every
> > > > > > > > > > > > > node
> > > > > > > > > > > > > >> and don't
> > > > > > > > > > > > > >> find any evidence of a serious performance
> > impact, and
> > > > > > users
> > > > > > > > > > > indicate
> > > > > > > > > > > > > they
> > > > > > > > > > > > > >> would
> > > > > > > > > > > > > >> like to see this metric at all nodes, then we
> can
> > > > easily
> > > > > > > start
> > > > > > > > > > > reporting
> > > > > > > > > > > > > >> them as well.
> > > > > > > > > > > > > >> WDYT?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> That said, sink nodes seem like a reasonable
> > > > exception to
> > > > > > > the
> > > > > > > > > > rule.
> > > > > > > > > > > > > >> Obviously users
> > > > > > > > > > > > > >> should be able to detect the time when the
> record
> > > > reaches
> > > > > > > the
> > > > > > > > > > output
> > > > > > > > > > > > > topic
> > > > > > > > > > > > > >> but that
> > > > > > > > > > > > > >> still leaves a gap in understanding how long the
> > > > > > production
> > > > > > > > > > latency
> > > > > > > > > > > was.
> > > > > > > > > > > > > >> This mirrors
> > > > > > > > > > > > > >> the consumption latency that is exposed by the
> > > > task-level
> > > > > > > > > metrics,
> > > > > > > > > > > which
> > > > > > > > > > > > > >> are measured
> > > > > > > > > > > > > >> at the source node. For good symmetry what if we
> > > > actually
> > > > > > > > expose
> > > > > > > > > > > both
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> source
> > > > > > > > > > > > > >> and sink latency at the task-level? ie report
> both
> > > > sets of
> > > > > > > > > > > statistical
> > > > > > > > > > > > > >> measurements with
> > > > > > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> @Bill
> > > > > > > > > > > > > >> Thanks for the comment about regarding the min!
> I
> > > > hadn't
> > > > > > > > > > considered
> > > > > > > > > > > that
> > > > > > > > > > > > > >> and it's
> > > > > > > > > > > > > >> quite useful to think about how and what is
> useful
> > > > from a
> > > > > > > > users
> > > > > > > > > > > point of
> > > > > > > > > > > > > >> view.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regarding your second. point, I'm inclined to
> > leave
> > > > that
> > > > > > as
> > > > > > > an
> > > > > > > > > > > > > >> implementation detail
> > > > > > > > > > > > > >> but my take would be that the user should be
> > allowed
> > > > to
> > > > > > > > control
> > > > > > > > > > the
> > > > > > > > > > > > > record
> > > > > > > > > > > > > >> timestamp
> > > > > > > > > > > > > >> used for this with the timestamp extractor. My
> > > > impression
> > > > > > is
> > > > > > > > > that
> > > > > > > > > > > users
> > > > > > > > > > > > > may
> > > > > > > > > > > > > >> often embed
> > > > > > > > > > > > > >> the actual event time in the payload for
> whatever
> > > > reason,
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > > > > >> represents the "true"
> > > > > > > > > > > > > >> timestamp as far as the Streams topology is
> > concerned.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > > > > > bbejeck@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a
> useful
> > > > metric
> > > > > > to
> > > > > > > > > add.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Regarding tracking min, I  think it could be
> > > > valuable for
> > > > > > > > users
> > > > > > > > > > to
> > > > > > > > > > > > > >> discern
> > > > > > > > > > > > > >>> which part of their topologies are more
> efficient
> > > > since
> > > > > > > this
> > > > > > > > > is a
> > > > > > > > > > > > > >>> task-level metric.  I realize everyone seems to
> > be on
> > > > > > board
> > > > > > > > > with
> > > > > > > > > > > > > >> including
> > > > > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on
> > this
> > > > topic
> > > > > > > > should
> > > > > > > > > > we
> > > > > > > > > > > > > decide
> > > > > > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> I do have a question regarding the calculation
> of
> > > > > > > staleness.
> > > > > > > > > > > > > >>> Is there going to be a consideration for
> > timestamp
> > > > > > > > extractors?
> > > > > > > > > > > Users
> > > > > > > > > > > > > >> could
> > > > > > > > > > > > > >>> prefer to use a timestamp embedded in the
> > payload,
> > > > and it
> > > > > > > > could
> > > > > > > > > > > skew
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >>> measurements.
> > > > > > > > > > > > > >>> I was wondering if we should specify in the KIP
> > if
> > > > > > setting
> > > > > > > > the
> > > > > > > > > > > arrival
> > > > > > > > > > > > > >> time
> > > > > > > > > > > > > >>> is always going to come from the record
> > timestamp,
> > > > or is
> > > > > > > this
> > > > > > > > > an
> > > > > > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > > >>> Bill
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J.
> Sax <
> > > > > > > > > > mjsax@apache.org>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> I think it's not useful to record the
> avg/mean;
> > it
> > > > > > > sensitive
> > > > > > > > > to
> > > > > > > > > > > > > >>>> outliers. We should rather track the median
> > (50th
> > > > > > > > percentile).
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Not sure if tracking min is useful, but I am
> > also
> > > > ok to
> > > > > > > > track
> > > > > > > > > > it.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> However, I find it odd to track 75th
> percentile.
> > > > > > Standard
> > > > > > > > > > measures
> > > > > > > > > > > > > >> would
> > > > > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need
> both,
> > so
> > > > maybe
> > > > > > > > > picking
> > > > > > > > > > > 90th
> > > > > > > > > > > > > >>>> might be more useful?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> About the name: "staleness" wound really odd,
> > and if
> > > > > > fact
> > > > > > > > the
> > > > > > > > > > > metric
> > > > > > > > > > > > > >>>> does capture "latency" so we should call it
> > > > "latency". I
> > > > > > > > > > > understand
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >>>> issue that we already have a latency metric.
> So
> > > > maybe we
> > > > > > > > could
> > > > > > > > > > > call it
> > > > > > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> While I agree that we should include
> > out-or-order
> > > > data
> > > > > > > (the
> > > > > > > > > KIP
> > > > > > > > > > > should
> > > > > > > > > > > > > >>>> talk about `out-of-order` data, not `late`
> data;
> > > > data is
> > > > > > > > only
> > > > > > > > > > > `late`
> > > > > > > > > > > > > if
> > > > > > > > > > > > > >>>> it's out-of-order and if it's dropped), I
> don't
> > > > really
> > > > > > > > > > understand
> > > > > > > > > > > why
> > > > > > > > > > > > > >>>> the new metric would help to configure grace
> > period
> > > > or
> > > > > > > > > retention
> > > > > > > > > > > time?
> > > > > > > > > > > > > >>>> As you mention in the KIP, both are define as
> > max
> > > > > > > difference
> > > > > > > > > of
> > > > > > > > > > > > > >>>> `event-time - stream-time` and thus the new
> > metric
> > > > that
> > > > > > > > takes
> > > > > > > > > > > > > >>>> system-/wallclock-time into account does not
> > seem to
> > > > > > help
> > > > > > > at
> > > > > > > > > > all.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to
> > Measure
> > > > > > > > Latency"
> > > > > > > > > by
> > > > > > > > > > > Gil
> > > > > > > > > > > > > >>>> Tene:
> > https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> -Matthias
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> It seems like there would still be plenty of
> > use
> > > > cases
> > > > > > > for
> > > > > > > > > > > recording
> > > > > > > > > > > > > >>>>> this metric at all processors and not just
> > stateful
> > > > > > ones,
> > > > > > > > but
> > > > > > > > > > I'm
> > > > > > > > > > > > > >> happy
> > > > > > > > > > > > > >>>>> to suspend my arguments for now. Since you're
> > > > proposing
> > > > > > > to
> > > > > > > > > keep
> > > > > > > > > > > > > >>>>> them at the processor-node level, it will be
> > > > seamless
> > > > > > > later
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > >>>>> in the stateless processors if we want. As a
> > wise
> > > > man
> > > > > > > once
> > > > > > > > > > said,
> > > > > > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Regarding the time measurement, it's an
> > > > implementation
> > > > > > > > detail
> > > > > > > > > > > > > >>>>> we don't need to consider in the KIP.
> > > > Nevertheless, I'd
> > > > > > > > > greatly
> > > > > > > > > > > > > >>>>> prefer to measure the system time again when
> > > > recording
> > > > > > > the
> > > > > > > > > > > > > >>>>> metric. I don't think we've seen any evidence
> > that
> > > > > > proves
> > > > > > > > > this
> > > > > > > > > > > > > >>>>> would harm performance, and the amount of
> > > > inaccuracy
> > > > > > > using
> > > > > > > > > > > > > >>>>> the cached system time could incur is
> actually
> > > > > > > substantial.
> > > > > > > > > > But,
> > > > > > > > > > > > > >>>>> if you want to just "not mention this" in the
> > KIP,
> > > > we
> > > > > > can
> > > > > > > > > defer
> > > > > > > > > > > to
> > > > > > > > > > > > > >>>>> the actual PR discussion, at which time we're
> > in a
> > > > > > better
> > > > > > > > > > > position
> > > > > > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Along the lines of the measurement accuracy
> > > > discussion,
> > > > > > > one
> > > > > > > > > > > > > >>>>> minor thought I had is that maybe we should
> > > > consider
> > > > > > > > > measuring
> > > > > > > > > > > > > >>>>> the task staleness metric at the sink, rather
> > than
> > > > the
> > > > > > > > > source,
> > > > > > > > > > so
> > > > > > > > > > > > > >> that
> > > > > > > > > > > > > >>>>> it includes the processing latency of the
> task
> > > > itself,
> > > > > > > not
> > > > > > > > > just
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>> latency
> > > > > > > > > > > > > >>>>> of everything up to, but not including, the
> > task
> > > > (which
> > > > > > > > seems
> > > > > > > > > > > > > >> confusing
> > > > > > > > > > > > > >>>>> for users). I guess this could also be an
> > > > > > implementation
> > > > > > > > > > detail,
> > > > > > > > > > > > > >>> though.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > > > > > >>>>> -John
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie
> > Blee-Goldman
> > > > > > > wrote:
> > > > > > > > > > > > > >>>>>> Hey all,
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> After discussing with Bruno I'd like to
> > propose a
> > > > > > small
> > > > > > > > > > > amendment,
> > > > > > > > > > > > > >>>>>> which is to record the processor-node-level
> > > > metrics
> > > > > > only
> > > > > > > > for
> > > > > > > > > > > > > >>> *stateful*
> > > > > > > > > > > > > >>>>>> *operators*. They would still be considered
> a
> > > > > > > > > > > "processor-node-level"
> > > > > > > > > > > > > >>>>>> metric and not a "state-store-level" metric
> > as the
> > > > > > > > staleness
> > > > > > > > > > is
> > > > > > > > > > > > > >> still
> > > > > > > > > > > > > >>>>>> a property of the node rather than of the
> > state
> > > > > > itself.
> > > > > > > > > > > However, it
> > > > > > > > > > > > > >>>> seems
> > > > > > > > > > > > > >>>>>> that this information is primarily useful
> for
> > > > stateful
> > > > > > > > > > operators
> > > > > > > > > > > > > >> that
> > > > > > > > > > > > > >>>> might
> > > > > > > > > > > > > >>>>>> be exposing state via IQ or otherwise
> > dependent
> > > > on the
> > > > > > > > > record
> > > > > > > > > > > time
> > > > > > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> It's worth calling out that recent
> performance
> > > > > > > > improvements
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>> metrics
> > > > > > > > > > > > > >>>>>> framework mean that we no longer fetch the
> > system
> > > > time
> > > > > > > at
> > > > > > > > > the
> > > > > > > > > > > > > >> operator
> > > > > > > > > > > > > >>>>>> level, but only once per task. In other
> words
> > the
> > > > > > system
> > > > > > > > > time
> > > > > > > > > > > is not
> > > > > > > > > > > > > >>>> updated
> > > > > > > > > > > > > >>>>>> between each process as a record flows
> > through the
> > > > > > > > > > subtopology,
> > > > > > > > > > > so
> > > > > > > > > > > > > >>>>>> debugging the processor-level latency via
> the
> > > > > > > stateleness
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > >> be
> > > > > > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > > > > > operator-level
> > > > > > > > > > metrics
> > > > > > > > > > > are
> > > > > > > > > > > > > >>> not
> > > > > > > > > > > > > >>>>>> *useful* relative to the task-level metric.
> > > > Upstream
> > > > > > > > caching
> > > > > > > > > > > and/or
> > > > > > > > > > > > > >>>>>> suppression
> > > > > > > > > > > > > >>>>>> can still cause a record's staleness at some
> > > > > > downstream
> > > > > > > > > > stateful
> > > > > > > > > > > > > >>>> operator
> > > > > > > > > > > > > >>>>>> to deviate from the task-level staleness
> > > > (recorded at
> > > > > > > the
> > > > > > > > > > source
> > > > > > > > > > > > > >>> node).
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> Please let me know if you have any concerns
> > about
> > > > this
> > > > > > > > > change.
> > > > > > > > > > > The
> > > > > > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno
> Cadonna
> > <
> > > > > > > > > > > bruno@confluent.io>
> > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > > > > > >>>>>>> I think we do not need it now. If we need
> > it, we
> > > > can
> > > > > > > add
> > > > > > > > > it.
> > > > > > > > > > > Here
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >>>>>>> old truism applies: Adding is always easier
> > than
> > > > > > > > removing.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > > > > > >>>>>>> I think it is good to have the staleness
> > metrics
> > > > also
> > > > > > > on
> > > > > > > > > > > > > >>>>>>> processor-node-level. If we do not want to
> > record
> > > > > > them
> > > > > > > on
> > > > > > > > > all
> > > > > > > > > > > > > >>>>>>> processor nodes, you could restrict the
> > > > recording to
> > > > > > > > > stateful
> > > > > > > > > > > > > >>>>>>> processor-nodes, since those are the ones
> > that
> > > > would
> > > > > > > > > benefit
> > > > > > > > > > > most
> > > > > > > > > > > > > >>> from
> > > > > > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> Best,
> > > > > > > > > > > > > >>>>>>> Bruno
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie
> > > > Blee-Goldman <
> > > > > > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to
> align
> > > > with the
> > > > > > > > > current
> > > > > > > > > > > > > >>> metrics.
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> Is it better to conform than to be right?
> > > > History
> > > > > > has
> > > > > > > a
> > > > > > > > > lot
> > > > > > > > > > > to say
> > > > > > > > > > > > > >>> on
> > > > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > > > >>>>>>>> matter
> > > > > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies to
> > the
> > > > fine
> > > > > > > > > details
> > > > > > > > > > of
> > > > > > > > > > > > > >>> metrics
> > > > > > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> More seriously, I figured if people are
> > looking
> > > > at
> > > > > > > this
> > > > > > > > > > metric
> > > > > > > > > > > > > >>> they're
> > > > > > > > > > > > > >>>>>>>> likely to
> > > > > > > > > > > > > >>>>>>>> be looking at all the others. Then naming
> > this
> > > > one
> > > > > > > > "-mean"
> > > > > > > > > > > would
> > > > > > > > > > > > > >>>> probably
> > > > > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg"
> > suffix in
> > > > the
> > > > > > > > other
> > > > > > > > > > > metrics
> > > > > > > > > > > > > >>> has
> > > > > > > > > > > > > >>>> a
> > > > > > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> As for the percentiles, I actually like
> p99
> > (and
> > > > > > p75)
> > > > > > > > > > better.
> > > > > > > > > > > I'll
> > > > > > > > > > > > > >>>> swap
> > > > > > > > > > > > > >>>>>>>> that out
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John
> > Roesler <
> > > > > > > > > > > vvcephei@apache.org
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is
> > there a
> > > > > > > reason
> > > > > > > > to
> > > > > > > > > > > choose
> > > > > > > > > > > > > >>>> "avg"
> > > > > > > > > > > > > >>>>>>>>> instead
> > > > > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid,
> and
> > I
> > > > might
> > > > > > be
> > > > > > > > > > > > > >> oversensitive
> > > > > > > > > > > > > >>>>>>> because
> > > > > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but
> it
> > > > strikes
> > > > > > me
> > > > > > > > > that
> > > > > > > > > > > "avg"
> > > > > > > > > > > > > >> is
> > > > > > > > > > > > > >>>>>>>>> actually
> > > > > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of
> > > > statistics,
> > > > > > > > > whereas
> > > > > > > > > > > "mean"
> > > > > > > > > > > > > >>> is
> > > > > > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but
> > none
> > > > with
> > > > > > > > > "mean";
> > > > > > > > > > > was
> > > > > > > > > > > > > >>> that
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Regarding the names of the percentile, I
> > > > actually
> > > > > > > > > couldn't
> > > > > > > > > > > find
> > > > > > > > > > > > > >>> _any_
> > > > > > > > > > > > > >>>>>>> other
> > > > > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there a
> > > > reason to
> > > > > > > > choose
> > > > > > > > > > > "99th"
> > > > > > > > > > > > > >> as
> > > > > > > > > > > > > >>>>>>> opposed
> > > > > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is
> not a
> > > > > > > criticism,
> > > > > > > > > I'm
> > > > > > > > > > > just
> > > > > > > > > > > > > >>>>>>> primarily
> > > > > > > > > > > > > >>>>>>>>> asking
> > > > > > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > > > > >>>>>>>>> -John
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie
> > > > Blee-Goldman
> > > > > > > > > wrote:
> > > > > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min
> > > > metric
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > sake of
> > > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the
> mean
> > (or
> > > > 50th
> > > > > > > > > > > percentile)
> > > > > > > > > > > > > >>> but I
> > > > > > > > > > > > > >>>>>>> can
> > > > > > > > > > > > > >>>>>>>>> see
> > > > > > > > > > > > > >>>>>>>>>> how users might expect it and find it a
> > bit
> > > > > > > > disorienting
> > > > > > > > > > > not to
> > > > > > > > > > > > > >>>>>>> have. So
> > > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > > > > > percentile)*
> > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > > > > > percentile)*
> > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John
> > Roesler <
> > > > > > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to
> > > > embarrass
> > > > > > > > > myself.
> > > > > > > > > > I
> > > > > > > > > > > > > >> guess
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>> mean
> > > > > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the
> > > > median, but
> > > > > > > > > neither
> > > > > > > > > > > are
> > > > > > > > > > > > > >> as
> > > > > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles
> > (99th
> > > > and
> > > > > > > max).
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any
> SLAs,
> > but
> > > > it
> > > > > > > does
> > > > > > > > > > round
> > > > > > > > > > > out
> > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > >>>>>>>>> mental
> > > > > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve
> always
> > > > graphed
> > > > > > > min
> > > > > > > > > > along
> > > > > > > > > > > with
> > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > > > >>>>>>>>>>> metrics to help me understand how fast
> > the
> > > > system
> > > > > > > can
> > > > > > > > > be,
> > > > > > > > > > > which
> > > > > > > > > > > > > >>>>>>> helps
> > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a
> > > > relatively
> > > > > > > > > > inexpensive
> > > > > > > > > > > > > >> metric
> > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just
> > throw
> > > > it in.
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> > > > > > Blee-Goldman
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > > > > > >>>>>>>>>>>> I was considering it as the
> "end-to-end
> > > > latency
> > > > > > > *up*
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if
> > that
> > > > drives
> > > > > > > the
> > > > > > > > > > point
> > > > > > > > > > > > > >> home
> > > > > > > > > > > > > >>>>>>>>> better.
> > > > > > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is
> > > > received by
> > > > > > > that
> > > > > > > > > > > task" --
> > > > > > > > > > > > > >>>>>>> will
> > > > > > > > > > > > > >>>>>>>>> update
> > > > > > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where
> > the
> > > > min
> > > > > > > would
> > > > > > > > > > > actually
> > > > > > > > > > > > > >> be
> > > > > > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I
> don't
> > > > feel
> > > > > > > > strongly
> > > > > > > > > > > that we
> > > > > > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the
> start.
> > Can
> > > > you
> > > > > > > > > > enlighten
> > > > > > > > > > > me
> > > > > > > > > > > > > >>>>>>> with an
> > > > > > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the
> > > > overhead
> > > > > > of
> > > > > > > > > > adding
> > > > > > > > > > > > > >>>>>>> multiple
> > > > > > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to
> indicate
> > > > what
> > > > > > kind
> > > > > > > > of
> > > > > > > > > > > > > >>> performance
> > > > > > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the
> > 50th
> > > > > > > percentile
> > > > > > > > > > > would be
> > > > > > > > > > > > > >>> the
> > > > > > > > > > > > > >>>>>>>>> median
> > > > > > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the
> mean
> > > > *and*
> > > > > > the
> > > > > > > > 50th
> > > > > > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John
> > > > Roesler <
> > > > > > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP,
> > Sophie,
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion
> so
> > > > far:
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter
> > whether
> > > > we
> > > > > > > report
> > > > > > > > > > > "spans"
> > > > > > > > > > > > > >> or
> > > > > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in
> practice,
> > > > some of
> > > > > > > the
> > > > > > > > > > spans
> > > > > > > > > > > are
> > > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like
> time
> > > > spent
> > > > > > > > waiting
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > > > > > >>>>>>>>>>>>> time from the event happening to the
> > ETL
> > > > > > producer
> > > > > > > > > > > choosing to
> > > > > > > > > > > > > >>>>>>> send
> > > > > > > > > > > > > >>>>>>>>> it,
> > > > > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive
> buffers,
> > > > etc.,
> > > > > > etc.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically
> > easier to
> > > > > > > compute
> > > > > > > > > > spans
> > > > > > > > > > > by
> > > > > > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute
> e2e
> > > > > > latencies
> > > > > > > > by
> > > > > > > > > > > adding
> > > > > > > > > > > > > >>>>>>> spans.
> > > > > > > > > > > > > >>>>>>>>> You
> > > > > > > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> > > > > > computation
> > > > > > > > from
> > > > > > > > > > e2e
> > > > > > > > > > > > > >>>>>>> always
> > > > > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers,
> > whereas
> > > > > > > computing
> > > > > > > > > e2e
> > > > > > > > > > > > > >> latency
> > > > > > > > > > > > > >>>>>>>>> from
> > > > > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans
> > > > leading
> > > > > > up
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > end
> > > > > > > > > > > > > >> you
> > > > > > > > > > > > > >>>>>>>>> care
> > > > > > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to
> > have
> > > > > > spans
> > > > > > > > when
> > > > > > > > > > > they
> > > > > > > > > > > > > >> are
> > > > > > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency
> > is a
> > > > more
> > > > > > > > > general
> > > > > > > > > > > > > >>>>>>> measurement
> > > > > > > > > > > > > >>>>>>>>>>>>> that basically every
> person/application
> > > > cares
> > > > > > > about
> > > > > > > > > and
> > > > > > > > > > > > > >> should
> > > > > > > > > > > > > >>>>>>> be
> > > > > > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide
> > more
> > > > > > value
> > > > > > > to
> > > > > > > > > > more
> > > > > > > > > > > > > >>>>>>> people if
> > > > > > > > > > > > > >>>>>>>>> we
> > > > > > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding
> > > > "record-staleness" as
> > > > > > a
> > > > > > > > > name,
> > > > > > > > > > I
> > > > > > > > > > > > > >> think
> > > > > > > > > > > > > >>>>>>> I
> > > > > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other
> peoples'
> > > > > > > intuition.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is
> > nice,
> > > > > > since
> > > > > > > > the
> > > > > > > > > > > inside
> > > > > > > > > > > > > >> of
> > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > >>>>>>>>> task
> > > > > > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of
> > latency
> > > > in
> > > > > > some
> > > > > > > > > > cases.
> > > > > > > > > > > > > >> Plus,
> > > > > > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really
> > wanted to
> > > > > > know
> > > > > > > > (for
> > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>> purposes
> > > > > > > > > > > > > >>>>>>>>> of
> > > > > > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes
> source
> > > > events
> > > > > > to
> > > > > > > > > "show
> > > > > > > > > > > up" at
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at
> every
> > > > > > processor
> > > > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at
> > the
> > > > node
> > > > > > > > level.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to
> > record
> > > > also.
> > > > > > > > Even
> > > > > > > > > if
> > > > > > > > > > > it's
> > > > > > > > > > > > > >> a
> > > > > > > > > > > > > >>>>>>> poor
> > > > > > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of
> > people
> > > > might
> > > > > > > > > expect
> > > > > > > > > > > to see
> > > > > > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is
> > there
> > > > a
> > > > > > > reason
> > > > > > > > > to
> > > > > > > > > > > leave
> > > > > > > > > > > > > >> it
> > > > > > > > > > > > > >>>>>>>>> off?
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > > > > > >>>>>>>>>>>>> I agree we should include late
> records
> > > > (though
> > > > > > > not
> > > > > > > > > the
> > > > > > > > > > > ones
> > > > > > > > > > > > > >> we
> > > > > > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there
> > are
> > > > > > > > legitimately
> > > > > > > > > > > some
> > > > > > > > > > > > > >>>>>>> records
> > > > > > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the
> > whole
> > > > > > point
> > > > > > > > of
> > > > > > > > > > > these
> > > > > > > > > > > > > >>>>>>> metrics.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any
> > other
> > > > > > > feedback,
> > > > > > > > > > other
> > > > > > > > > > > > > >> than
> > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58,
> > Guozhang
> > > > Wang
> > > > > > > > wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few
> quick
> > > > > > thoughts:
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes
> > both
> > > > the
> > > > > > > > > processing
> > > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in
> > > > intermediate
> > > > > > > > > topics.
> > > > > > > > > > I
> > > > > > > > > > > have
> > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above
> > that the
> > > > > > > latency
> > > > > > > > > > > metric of
> > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > >>>>>>>>> task A
> > > > > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > > > > > > sub-topology
> > > > > > > > > > up-to
> > > > > > > > > > > but
> > > > > > > > > > > > > >>>>>>> not
> > > > > > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit
> > weird.
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the
> > name
> > > > > > > "latency"
> > > > > > > > > > > itself,
> > > > > > > > > > > > > >>>>>>> since
> > > > > > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > > > > > >>>>>>>>>>>>>> already have several "latency"
> metrics
> > > > already
> > > > > > > > which
> > > > > > > > > > are
> > > > > > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a record
> /
> > etc,
> > > > > > while
> > > > > > > > > here
> > > > > > > > > > > we are
> > > > > > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the
> record
> > > > > > timestamp.
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > > > > > > "record-staleness"
> > > > > > > > > > > > > >>>>>>> (note we
> > > > > > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in
> > which
> > > > > > case
> > > > > > > > > > > recording at
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>>>>> system-time before we start
> > processing the
> > > > > > > record
> > > > > > > > > > sounds
> > > > > > > > > > > > > >> more
> > > > > > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering
> > if the
> > > > > > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that
> we
> > > > already
> > > > > > > > have a
> > > > > > > > > > > > > >>>>>>> task-level
> > > > > > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a
> specific
> > > > node's
> > > > > > e2e
> > > > > > > > > > > latency is
> > > > > > > > > > > > > >>>>>>>>> similar
> > > > > > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level
> > > > processing
> > > > > > > > > > latency.
> > > > > > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness
> > > > metric is
> > > > > > > > > > > sufficient.
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM
> > Sophie
> > > > > > > > > Blee-Goldman <
> > > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a
> > particularly
> > > > > > > useful
> > > > > > > > > > gauge
> > > > > > > > > > > for
> > > > > > > > > > > > > >>>>>>> this
> > > > > > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at
> > putting an
> > > > > > > *upper
> > > > > > > > > > > *bound on
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose
> > > > percentiles
> > > > > > > > that
> > > > > > > > > > > would
> > > > > > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will
> > > > experience,
> > > > > > > and
> > > > > > > > > > what
> > > > > > > > > > > > > >>>>>>> *close
> > > > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these
> > specific
> > > > > > > numbers
> > > > > > > > > and
> > > > > > > > > > > > > >>>>>>> could be
> > > > > > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in
> > hearing
> > > > > > from
> > > > > > > > > users
> > > > > > > > > > on
> > > > > > > > > > > > > >>>>>>> this.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> > > > > > "hop-to-hop
> > > > > > > > > > > latency" in
> > > > > > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> > > > > > subtracting
> > > > > > > > the
> > > > > > > > > > > > > >>>>>>> previous
> > > > > > > > > > > > > >>>>>>>>> node's
> > > > > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we
> could
> > do
> > > > it
> > > > > > > either
> > > > > > > > > way
> > > > > > > > > > > since
> > > > > > > > > > > > > >>>>>>>>> you can
> > > > > > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I
> > think
> > > > the
> > > > > > > > > > end-to-end
> > > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is
> > not
> > > > to
> > > > > > > debug
> > > > > > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > > > > > >>>>>>>>> in the
> > > > > > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long
> it
> > > > takes
> > > > > > > > arecord
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this
> > might
> > > > be
> > > > > > > useful
> > > > > > > > > for
> > > > > > > > > > > users
> > > > > > > > > > > > > >>>>>>>>> who are
> > > > > > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just
> > > > produced
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > >>>>>>> included
> > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side
> effect
> > --
> > > > but
> > > > > > > > maybe I
> > > > > > > > > > > didn't
> > > > > > > > > > > > > >>>>>>> make
> > > > > > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address
> > this
> > > > in
> > > > > > the
> > > > > > > > KIP.
> > > > > > > > > > The
> > > > > > > > > > > > > >>>>>>> short
> > > > > > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I
> > added a
> > > > > > > paragraph
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>> end
> > > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the
> > reasoning
> > > > > > here,
> > > > > > > > > please
> > > > > > > > > > > let
> > > > > > > > > > > > > >>>>>>> me
> > > > > > > > > > > > > >>>>>>>>> know
> > > > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the
> > > > existing
> > > > > > > > metric
> > > > > > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > > > > > Node#process
> > > > > > > > > method
> > > > > > > > > > > to
> > > > > > > > > > > > > >>>>>>> run
> > > > > > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative
> to
> > the
> > > > > > event
> > > > > > > > > > > timestamp.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision
> > there is
> > > > > > pretty
> > > > > > > > > > > confusing
> > > > > > > > > > > > > >>>>>>> so
> > > > > > > > > > > > > >>>>>>>>> I've
> > > > > > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to
> > > > "end-to-end-latency"
> > > > > > > > which I
> > > > > > > > > > > feel
> > > > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM
> > Boyang
> > > > Chen
> > > > > > <
> > > > > > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting
> > the
> > > > E2E
> > > > > > > > latency
> > > > > > > > > > is
> > > > > > > > > > > > > >>>>>>>>> important
> > > > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of
> the
> > > > > > > application.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational
> of
> > > > picking
> > > > > > > 75,
> > > > > > > > > 99
> > > > > > > > > > > and
> > > > > > > > > > > > > >>>>>>> max
> > > > > > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use
> > 50, 90
> > > > > > > > > percentile
> > > > > > > > > > as
> > > > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being
> > computed is
> > > > > > > > > cumulative,
> > > > > > > > > > > I.E
> > > > > > > > > > > > > >>>>>>> if a
> > > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) =
> > > > T(B->C) +
> > > > > > > P(B)
> > > > > > > > =
> > > > > > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents
> the
> > > > captured
> > > > > > > > > > latency,
> > > > > > > > > > > > > >>>>>>> and T()
> > > > > > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the
> records
> > > > between
> > > > > > > two
> > > > > > > > > > nodes,
> > > > > > > > > > > > > >>>>>>>>> including
> > > > > > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose,
> maybe
> > > > having
> > > > > > > > T(B->C)
> > > > > > > > > > and
> > > > > > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop
> > latency",
> > > > > > > > otherwise
> > > > > > > > > > if
> > > > > > > > > > > > > >>>>>>> there
> > > > > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are
> > > > affected in
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > > >>>>>>> time.
> > > > > > > > > > > > > >>>>>>>>> In the
> > > > > > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is
> > meaningful
> > > > only
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > > >>>>>>> record
> > > > > > > > > > > > > >>>>>>>>> exits
> > > > > > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time
> > this
> > > > > > record
> > > > > > > > > spent
> > > > > > > > > > > > > >>>>>>> inside
> > > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate
> > > > treatment
> > > > > > for
> > > > > > > > > sink
> > > > > > > > > > > > > >>>>>>> nodes and
> > > > > > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only
> > count
> > > > the
> > > > > > time
> > > > > > > > > > > receiving
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a
> > solution
> > > > here,
> > > > > > > > just
> > > > > > > > > > > want to
> > > > > > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is
> > reasonable.
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late
> > > > arrival
> > > > > > > > records
> > > > > > > > > > as
> > > > > > > > > > > > > >>>>>>> well,
> > > > > > > > > > > > > >>>>>>>>> they
> > > > > > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs
> > when the
> > > > > > > > > > out-of-order
> > > > > > > > > > > > > >>>>>>>>> records are
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records.
> > > > Should we
> > > > > > > > also
> > > > > > > > > > > supply
> > > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user
> > should
> > > > just
> > > > > > > take
> > > > > > > > > > care
> > > > > > > > > > > of
> > > > > > > > > > > > > >>>>>>> it
> > > > > > > > > > > > > >>>>>>>>> by
> > > > > > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we
> > > > haven't
> > > > > > > > > discussed
> > > > > > > > > > > its
> > > > > > > > > > > > > >>>>>>>>> relation
> > > > > > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency
> metrics,
> > > > could
> > > > > > you
> > > > > > > > add
> > > > > > > > > > > some
> > > > > > > > > > > > > >>>>>>>>> context
> > > > > > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to
> use
> > > > which`
> > > > > > > > > tutorial
> > > > > > > > > > > for
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM
> > Sophie
> > > > > > > > > Blee-Goldman
> > > > > > > > > > <
> > > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion
> on
> > > > KIP-613
> > > > > > > > which
> > > > > > > > > > aims
> > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > >>>>>>>>> add
> > > > > > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams.
> Please
> > > > take a
> > > > > > > look:
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
@Bruno
No, this would be after inserting in the cache/state store/etc -- basically
once
all operations have completed such that the record would be present if
queried after that time.

@Guozhang
I'm glad you suggested the TRACE level. It seems we've been
heading in that direction for a while so I'm happy to add it as part of
this KIP. Especially if it unblocks the question of whether to update the
system time (for now).

On Mon, May 18, 2020 at 1:09 AM Bruno Cadonna <br...@confluent.io> wrote:

> Hi Sophie,
>
> Just a question for my understanding. When you say " t_A be the time
> when it is finished being processed by the aggregator node", this
> means before caching (or putting into the state store if caching is
> disabled), right?
>
> Best,
> Bruno
>
> On Sat, May 16, 2020 at 6:57 PM Guozhang Wang <wa...@gmail.com> wrote:
> >
> > We do have a big JIRA for making sensor recording to be sufficient. But
> it
> > is a bit hard to try to reduce the num.system clock calls in general (but
> > admittedly, in nowadays OS they are supposed to be very cheap as well),
> > I've just recently merged a small PR trying to make less
> > SystemTime.currentMilliseconds calls whenever possible, but it is hard to
> > enforce that for all callers.
> >
> > Guozhang
> >
> > On Sat, May 16, 2020 at 9:46 AM John Roesler <vv...@apache.org>
> wrote:
> >
> > > FWIW, I’d be fine with TRACE, if it makes the processor-level metrics
> more
> > > palatable to everyone.
> > >
> > > Ultimately, I think this is just kicking the can down the road, and we
> > > should really just refactor our sensors/metrics to be lower overhead.
> But
> > > that’s too much work to consider right before the release so for now,
> we
> > > can kick that can just a little farther.
> > >
> > > Thanks for proposing a solution, Guozhang!
> > >
> > > -John
> > >
> > > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> > > > Cool, that's good points -- we should not let our current metrics
> > > proposal
> > > > to depend on future roadmaps, since they can still be removed later
> if
> > > ever
> > > > becoming not valuable after all.
> > > >
> > > > On the other hand, what do you think about introducing TRACE level
> > > metrics
> > > > recording and make the the processor-node level metrics on TRACE?
> > > >
> > > > Guozhang
> > > >
> > > > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <
> sophie@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Yeah, I had proposed to replace the "source + sink" task-level
> metric
> > > > > with "sink only" task-level metrics plus the stateful node-level
> ones.
> > > > >
> > > > > I completely agree that this proposal should and would look
> different
> > > > > once we complete those two proposals, ie decouple caching and
> > > > > consolidate suppression. But as it is right now, caching and
> > > suppression
> > > > > cause the actual time-of-processing of records to vary greatly
> > > throughout
> > > > > a subtopology such that the delta between a source node and an
> > > intermediate
> > > > > one is potentially quite large.
> > > > >
> > > > > For this exact reason I would argue that this metric is still
> valuable
> > > at
> > > > > the node-level
> > > > >  even if the "current time" is actually only updated once for a
> task.
> > > > > Obviously this
> > > > > would not convey any information about the actual processing
> latency,
> > > but
> > > > > as you
> > > > > stated, this is likely to be a small delta on the scale of
> > > milliseconds.
> > > > > But the latency
> > > > > due to a suppression can be on the order of minutes, hours, or even
> > > days!
> > > > > If
> > > > > there is more than one suppression or caching in a subtopology, the
> > > source
> > > > > and sink
> > > > > e2e latency could be wildly misrepresentative of the e2e latency
> of a
> > > node
> > > > > in the middle.
> > > > >
> > > > > Anyways, I removed this last point from the current KIP document
> > > because I
> > > > > felt
> > > > > it was ultimately an implementation detail. But the larger point I
> > > would
> > > > > argue is that:
> > > > > if we are truly worried about the performance hit of fetching the
> > > system
> > > > > time at every
> > > > > node, we should just not fetch the system time at every node
> instead of
> > > > > dropping the
> > > > > metric altogether. It may be slightly off but only on the order of
> ms
> > > and
> > > > > not on the order
> > > > > of hours, and it's still more useful than no metric at all
> > > > >
> > > > >
> > > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Sophie,
> > > > > >
> > > > > > I think your motivation makes sense, in fact before KIP-444 we
> used
> > > to
> > > > > have
> > > > > > processor-node-level process latency as DEBUG which would
> > > holistically
> > > > > > address the motivation but we removed it since it is indeed quite
> > > > > expensive
> > > > > > (I do not have concrete numbers at hand, but basically you'd
> have to
> > > call
> > > > > > system-wall-clock time a couple of time for each record, each
> > > processor
> > > > > > node), and in the long run I feel such scenarios would be less
> > > common if
> > > > > we
> > > > > > 1) decouple caching from emitting and 2) always put suppression
> right
> > > > > > before sink only, and that's also why I've proposed to remove
> them in
> > > > > > KIP-444.
> > > > > >
> > > > > > I missed an update in your wiki that right now you've updated the
> > > > > > task-level metric to be recorded at the sink node, and thought
> it is
> > > > > still
> > > > > > at the source node, but assuming it is still the case, then
> > > practically
> > > > > any
> > > > > > state store's staleness of the task could roughly be measured as
> that
> > > > > > task-level staleness plus a small delta (usually tens of mills),
> and
> > > the
> > > > > > down stream task B's staleness - up stream task A's staleness
> would
> > > > > capture
> > > > > > the suppression effect at the end of A plus the intermediate
> topic
> > > > > > producing -> consuming latency, which usually would not require
> to be
> > > > > > distinguished.
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <
> > > sophie@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Guozhang,
> > > > > > >
> > > > > > > Thanks for the response. I meant to ask this earlier and
> forgot,
> > > but do
> > > > > > you
> > > > > > > have any data or benchmarking results on hand for the
> performance
> > > hit
> > > > > we
> > > > > > > took due to the metrics? Or maybe better yet, on the
> performance we
> > > > > > gained
> > > > > > > back by dropping some metrics? I think it would help us to
> better
> > > > > > > understand
> > > > > > > the tradeoffs with some concrete numbers to compare.
> > > > > > >
> > > > > > > I guess I'm inclined to want to include these metrics at the
> > > operator
> > > > > > level
> > > > > > > on
> > > > > > > DEBUG since, in my mind, that's exactly what DEBUG is for:
> > > performance-
> > > > > > > focused users will presumably run at INFO level and even those
> who
> > > do
> > > > > > > use DEBUG may only do so temporarily, when they actually have
> > > something
> > > > > > > to debug.
> > > > > > >
> > > > > > > Of course, I'm not running a Streams app myself so all this is
> just
> > > > > > > conjecture.
> > > > > > > But it seems reasonable to interpret the two metrics levels as
> > > "minimal
> > > > > > > useful
> > > > > > > information" and "all information that provides insight into
> the
> > > > > system"
> > > > > > >
> > > > > > > If we can agree there, the question becomes "does this provide
> any
> > > > > > > additional
> > > > > > > insight?" I would still argue that it does. To give a concrete
> > > example,
> > > > > > > let's say
> > > > > > > the user has a basic
> > > > > > >
> > > > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > > > > >
> > > > > > > subtopology, the application performing IQ on the transformer's
> > > state.
> > > > > > The
> > > > > > > user
> > > > > > > wants to know how long it's taking records to be reflected in
> the
> > > > > results
> > > > > > > of a query.
> > > > > > > With only task-level metrics, they can only guess based on the
> e2e
> > > > > > latency
> > > > > > > of the
> > > > > > > previous task and of this one. But neither of those is a good
> > > > > > > representation of the
> > > > > > > time it will take a record to be processed by the transformer:
> the
> > > > > > previous
> > > > > > > task's
> > > > > > > latency will not include the time for the record to go through
> the
> > > > > > > repartitioning,
> > > > > > > and the current task's latency is measured after a
> suppression. The
> > > > > > > suppression
> > > > > > > itself will introduce significant latency and doesn't provide
> an
> > > > > accurate
> > > > > > > estimate.
> > > > > > > Caching would also do what it does best, messing with our
> > > understanding
> > > > > > of
> > > > > > > time :)
> > > > > > >
> > > > > > > It also means you can't measure the consumption latency, so
> there
> > > would
> > > > > > be
> > > > > > > no way to tell if the e2e latency is so high because the very
> first
> > > > > > > subtopology is
> > > > > > > extremely slow, or the record was severely delayed in reaching
> the
> > > > > input
> > > > > > > topic.
> > > > > > > I guess my earlier proposal to include both source and sink
> metrics
> > > > > could
> > > > > > > address this issue, but it seems to solve the problem more
> > > holistically
> > > > > > --
> > > > > > > and for
> > > > > > > more complex subtopologies -- by including metrics at the
> processor
> > > > > level
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <
> wangguoz@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hey folks,
> > > > > > > >
> > > > > > > > I want to make a final fight on processor-node-level metrics
> :)
> > > Post
> > > > > > > > KIP-444 we've actually removed a lot of node-level metrics
> since
> > > it
> > > > > is
> > > > > > > too
> > > > > > > > expensive to record at that level and their values are not
> proven
> > > > > worth
> > > > > > > it.
> > > > > > > > Again I'd use the "we have to either enable all or none of
> DEBUG
> > > > > > metrics"
> > > > > > > > card, e.g. if we want to look into per-task process-latency
> > > which is
> > > > > > > DEBUG,
> > > > > > > > we'd have to record all process-node / state-store / cache
> level
> > > > > > metrics.
> > > > > > > > On the other hand, within a task we usually do not have a
> lot of
> > > > > > stateful
> > > > > > > > operator nodes and if we want to find if certain stateful
> nodes
> > > are
> > > > > the
> > > > > > > > bottleneck of process latency, that is usually discoverable
> from
> > > the
> > > > > > > > state-store level read / write latencies already. So I'd
> imagine
> > > that
> > > > > > > > suppose there's a sub-topology with multiple state stores in
> it,
> > > if
> > > > > we
> > > > > > > > found the task-level process latency is high, we can still
> tell
> > > which
> > > > > > > state
> > > > > > > > stores within it is the main bottleneck from the state store
> > > read /
> > > > > > write
> > > > > > > > latency (I'm assuming the IO latency is always dominant
> compared
> > > with
> > > > > > > > others).
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > > > > sophie@confluent.io
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > @Matthias
> > > > > > > > > > And that's what he says about the 95th percentile!
> Imagine
> > > what
> > > > > he
> > > > > > > > would
> > > > > > > > > say about the 50th :P
> > > > > > > > >
> > > > > > > > > I should have kept watching another 10 minutes. He gets
> around
> > > to
> > > > > > > > covering
> > > > > > > > > the 50th, and let's
> > > > > > > > > just say he is not a fan:
> https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <
> > > vvcephei@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, Sophie!
> > > > > > > > > >
> > > > > > > > > > I think this makes perfect sense. It will be much more
> > > intuitive
> > > > > to
> > > > > > > use
> > > > > > > > > > the metric for the stated motivation this way.  I’d be in
> > > favor
> > > > > of
> > > > > > > the
> > > > > > > > > > proposal after this update.
> > > > > > > > > >
> > > > > > > > > > Thanks again for taking this on,
> > > > > > > > > > -John
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman
> wrote:
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to clarify/modify one aspect of this KIP,
> which
> > > is to
> > > > > > > record
> > > > > > > > > the
> > > > > > > > > > > staleness
> > > > > > > > > > > at the *completion* of the record's processing by the
> > > > > > operator/task
> > > > > > > > in
> > > > > > > > > > > question,
> > > > > > > > > > > rather than on the intake. The task-level metrics will
> be
> > > > > > recorded
> > > > > > > at
> > > > > > > > > the
> > > > > > > > > > > sink
> > > > > > > > > > > node instead of at the source, and the operator-level
> > > metrics
> > > > > > will
> > > > > > > be
> > > > > > > > > > > recorded
> > > > > > > > > > > at the end of the operation.
> > > > > > > > > > >
> > > > > > > > > > > The stated purpose and intended usefulness of this KIP
> is
> > > to
> > > > > give
> > > > > > > > > users a
> > > > > > > > > > > way
> > > > > > > > > > > to gauge roughly how long it takes for a record to be
> > > reflected
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > "results",
> > > > > > > > > > > whether these results are being read from an output
> topic
> > > or
> > > > > > > through
> > > > > > > > > IQ.
> > > > > > > > > > To
> > > > > > > > > > > take the IQ example, the results of a record are
> obviously
> > > not
> > > > > > > > visible
> > > > > > > > > > until
> > > > > > > > > > > *after* that node has finished processing it. The
> > > staleness,
> > > > > > while
> > > > > > > > > still
> > > > > > > > > > > potentially
> > > > > > > > > > > useful as it can impact the *way* a record is
> processed in
> > > a
> > > > > > > stateful
> > > > > > > > > and
> > > > > > > > > > > time-
> > > > > > > > > > > dependent operator, is not part of the problem this KIP
> > > > > > > specifically
> > > > > > > > > set
> > > > > > > > > > out
> > > > > > > > > > > to solve.
> > > > > > > > > > >
> > > > > > > > > > > In light of this I think it's appropriate to revert the
> > > name
> > > > > > change
> > > > > > > > > back
> > > > > > > > > > to
> > > > > > > > > > > include
> > > > > > > > > > > latency, since "staleness" as described above only
> makes
> > > sense
> > > > > > when
> > > > > > > > > > > measuring
> > > > > > > > > > > relative to the arrival of a record at a task/node. I'd
> > > propose
> > > > > > to
> > > > > > > > save
> > > > > > > > > > the
> > > > > > > > > > > term
> > > > > > > > > > > "staleness" for that particular meaning, and adopt
> > > Matthias's
> > > > > > > > > suggestion
> > > > > > > > > > of
> > > > > > > > > > > "record-e2e-latency" for this.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for hanging in there all. Please let me know if
> you
> > > have
> > > > > > any
> > > > > > > > > > > concerns about
> > > > > > > > > > > this change!
> > > > > > > > > > >
> > > > > > > > > > > Sophie
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > > > > mjsax@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I am also happy with max/min/99/90. And I buy your
> naming
> > > > > > > argument
> > > > > > > > > > about
> > > > > > > > > > > > staleness vs latency.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > -Matthias
> > > > > > > > > > > >
> > > > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > > > > Hey Sophie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie
> Blee-Goldman <
> > > > > > > > > > > > sophie@confluent.io>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> @Matthias
> > > > > > > > > > > > >> Regarding tracking the 50th percentile, I'll refer
> > > you to
> > > > > > the
> > > > > > > > 4:53
> > > > > > > > > > mark
> > > > > > > > > > > > of
> > > > > > > > > > > > >> the video
> > > > > > > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> And that's what he says about the 95th percentile!
> > > Imagine
> > > > > > > what
> > > > > > > > he
> > > > > > > > > > would
> > > > > > > > > > > > >> say about
> > > > > > > > > > > > >> the 50th :P
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> But seriously, since we can't seem to agree that
> the
> > > mean
> > > > > or
> > > > > > > > 50th
> > > > > > > > > > > > >> percentile is actually
> > > > > > > > > > > > >> useful I'm inclined to resurrect my original
> proposal,
> > > > > > > neither.
> > > > > > > > > But
> > > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > >> that's a good
> > > > > > > > > > > > >> argument against the 75th, which I admittedly
> chose
> > > > > somewhat
> > > > > > > > > > > > arbitrarily as
> > > > > > > > > > > > >> an
> > > > > > > > > > > > >> intermediate between the 50th and the higher
> > > percentiles.
> > > > > > How
> > > > > > > > > about:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -max
> > > > > > > > > > > > >> -p99
> > > > > > > > > > > > >> -p90
> > > > > > > > > > > > >> -min
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> with p50/mean still up for debate if anyone feels
> > > strongly
> > > > > > for
> > > > > > > > > > either of
> > > > > > > > > > > > >> them.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regarding the name, I've already flip-flopped on
> this
> > > so
> > > > > I'm
> > > > > > > > > > definitely
> > > > > > > > > > > > >> still open to
> > > > > > > > > > > > >> further arguments. But the reason for changing it
> from
> > > > > > > > > > > > end-to-end-latency
> > > > > > > > > > > > >> (which
> > > > > > > > > > > > >> is similar to what you propose) is that this
> metric
> > > > > > > technically
> > > > > > > > > > reflects
> > > > > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > > > > >> the record is when it's *received* by the
> operator,
> > > not
> > > > > when
> > > > > > > > it's
> > > > > > > > > > > > processed
> > > > > > > > > > > > >> by the operator.
> > > > > > > > > > > > >> It seemed like there was the potential for
> confusion
> > > that
> > > > > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > > > > >> represent the latency from the event creation to
> the
> > > time
> > > > > > the
> > > > > > > > > > processor
> > > > > > > > > > > > is
> > > > > > > > > > > > >> done
> > > > > > > > > > > > >> processing it.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> @John
> > > > > > > > > > > > >> I'd rather err on the side of "not-enough"
> metrics as
> > > we
> > > > > can
> > > > > > > > > always
> > > > > > > > > > add
> > > > > > > > > > > > >> this to the
> > > > > > > > > > > > >> stateless metrics later on. If we decide to
> measure
> > > the
> > > > > time
> > > > > > > at
> > > > > > > > > > every
> > > > > > > > > > > > node
> > > > > > > > > > > > >> and don't
> > > > > > > > > > > > >> find any evidence of a serious performance
> impact, and
> > > > > users
> > > > > > > > > > indicate
> > > > > > > > > > > > they
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >> like to see this metric at all nodes, then we can
> > > easily
> > > > > > start
> > > > > > > > > > reporting
> > > > > > > > > > > > >> them as well.
> > > > > > > > > > > > >> WDYT?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> That said, sink nodes seem like a reasonable
> > > exception to
> > > > > > the
> > > > > > > > > rule.
> > > > > > > > > > > > >> Obviously users
> > > > > > > > > > > > >> should be able to detect the time when the record
> > > reaches
> > > > > > the
> > > > > > > > > output
> > > > > > > > > > > > topic
> > > > > > > > > > > > >> but that
> > > > > > > > > > > > >> still leaves a gap in understanding how long the
> > > > > production
> > > > > > > > > latency
> > > > > > > > > > was.
> > > > > > > > > > > > >> This mirrors
> > > > > > > > > > > > >> the consumption latency that is exposed by the
> > > task-level
> > > > > > > > metrics,
> > > > > > > > > > which
> > > > > > > > > > > > >> are measured
> > > > > > > > > > > > >> at the source node. For good symmetry what if we
> > > actually
> > > > > > > expose
> > > > > > > > > > both
> > > > > > > > > > > > the
> > > > > > > > > > > > >> source
> > > > > > > > > > > > >> and sink latency at the task-level? ie report both
> > > sets of
> > > > > > > > > > statistical
> > > > > > > > > > > > >> measurements with
> > > > > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> @Bill
> > > > > > > > > > > > >> Thanks for the comment about regarding the min! I
> > > hadn't
> > > > > > > > > considered
> > > > > > > > > > that
> > > > > > > > > > > > >> and it's
> > > > > > > > > > > > >> quite useful to think about how and what is useful
> > > from a
> > > > > > > users
> > > > > > > > > > point of
> > > > > > > > > > > > >> view.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regarding your second. point, I'm inclined to
> leave
> > > that
> > > > > as
> > > > > > an
> > > > > > > > > > > > >> implementation detail
> > > > > > > > > > > > >> but my take would be that the user should be
> allowed
> > > to
> > > > > > > control
> > > > > > > > > the
> > > > > > > > > > > > record
> > > > > > > > > > > > >> timestamp
> > > > > > > > > > > > >> used for this with the timestamp extractor. My
> > > impression
> > > > > is
> > > > > > > > that
> > > > > > > > > > users
> > > > > > > > > > > > may
> > > > > > > > > > > > >> often embed
> > > > > > > > > > > > >> the actual event time in the payload for whatever
> > > reason,
> > > > > > and
> > > > > > > > this
> > > > > > > > > > > > >> represents the "true"
> > > > > > > > > > > > >> timestamp as far as the Streams topology is
> concerned.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > > > > bbejeck@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful
> > > metric
> > > > > to
> > > > > > > > add.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Regarding tracking min, I  think it could be
> > > valuable for
> > > > > > > users
> > > > > > > > > to
> > > > > > > > > > > > >> discern
> > > > > > > > > > > > >>> which part of their topologies are more efficient
> > > since
> > > > > > this
> > > > > > > > is a
> > > > > > > > > > > > >>> task-level metric.  I realize everyone seems to
> be on
> > > > > board
> > > > > > > > with
> > > > > > > > > > > > >> including
> > > > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on
> this
> > > topic
> > > > > > > should
> > > > > > > > > we
> > > > > > > > > > > > decide
> > > > > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> I do have a question regarding the calculation of
> > > > > > staleness.
> > > > > > > > > > > > >>> Is there going to be a consideration for
> timestamp
> > > > > > > extractors?
> > > > > > > > > > Users
> > > > > > > > > > > > >> could
> > > > > > > > > > > > >>> prefer to use a timestamp embedded in the
> payload,
> > > and it
> > > > > > > could
> > > > > > > > > > skew
> > > > > > > > > > > > the
> > > > > > > > > > > > >>> measurements.
> > > > > > > > > > > > >>> I was wondering if we should specify in the KIP
> if
> > > > > setting
> > > > > > > the
> > > > > > > > > > arrival
> > > > > > > > > > > > >> time
> > > > > > > > > > > > >>> is always going to come from the record
> timestamp,
> > > or is
> > > > > > this
> > > > > > > > an
> > > > > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > >>> Bill
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > > > > > > mjsax@apache.org>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> I think it's not useful to record the avg/mean;
> it
> > > > > > sensitive
> > > > > > > > to
> > > > > > > > > > > > >>>> outliers. We should rather track the median
> (50th
> > > > > > > percentile).
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Not sure if tracking min is useful, but I am
> also
> > > ok to
> > > > > > > track
> > > > > > > > > it.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> However, I find it odd to track 75th percentile.
> > > > > Standard
> > > > > > > > > measures
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need both,
> so
> > > maybe
> > > > > > > > picking
> > > > > > > > > > 90th
> > > > > > > > > > > > >>>> might be more useful?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> About the name: "staleness" wound really odd,
> and if
> > > > > fact
> > > > > > > the
> > > > > > > > > > metric
> > > > > > > > > > > > >>>> does capture "latency" so we should call it
> > > "latency". I
> > > > > > > > > > understand
> > > > > > > > > > > > the
> > > > > > > > > > > > >>>> issue that we already have a latency metric. So
> > > maybe we
> > > > > > > could
> > > > > > > > > > call it
> > > > > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> While I agree that we should include
> out-or-order
> > > data
> > > > > > (the
> > > > > > > > KIP
> > > > > > > > > > should
> > > > > > > > > > > > >>>> talk about `out-of-order` data, not `late` data;
> > > data is
> > > > > > > only
> > > > > > > > > > `late`
> > > > > > > > > > > > if
> > > > > > > > > > > > >>>> it's out-of-order and if it's dropped), I don't
> > > really
> > > > > > > > > understand
> > > > > > > > > > why
> > > > > > > > > > > > >>>> the new metric would help to configure grace
> period
> > > or
> > > > > > > > retention
> > > > > > > > > > time?
> > > > > > > > > > > > >>>> As you mention in the KIP, both are define as
> max
> > > > > > difference
> > > > > > > > of
> > > > > > > > > > > > >>>> `event-time - stream-time` and thus the new
> metric
> > > that
> > > > > > > takes
> > > > > > > > > > > > >>>> system-/wallclock-time into account does not
> seem to
> > > > > help
> > > > > > at
> > > > > > > > > all.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to
> Measure
> > > > > > > Latency"
> > > > > > > > by
> > > > > > > > > > Gil
> > > > > > > > > > > > >>>> Tene:
> https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> -Matthias
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> It seems like there would still be plenty of
> use
> > > cases
> > > > > > for
> > > > > > > > > > recording
> > > > > > > > > > > > >>>>> this metric at all processors and not just
> stateful
> > > > > ones,
> > > > > > > but
> > > > > > > > > I'm
> > > > > > > > > > > > >> happy
> > > > > > > > > > > > >>>>> to suspend my arguments for now. Since you're
> > > proposing
> > > > > > to
> > > > > > > > keep
> > > > > > > > > > > > >>>>> them at the processor-node level, it will be
> > > seamless
> > > > > > later
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > >>>>> in the stateless processors if we want. As a
> wise
> > > man
> > > > > > once
> > > > > > > > > said,
> > > > > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Regarding the time measurement, it's an
> > > implementation
> > > > > > > detail
> > > > > > > > > > > > >>>>> we don't need to consider in the KIP.
> > > Nevertheless, I'd
> > > > > > > > greatly
> > > > > > > > > > > > >>>>> prefer to measure the system time again when
> > > recording
> > > > > > the
> > > > > > > > > > > > >>>>> metric. I don't think we've seen any evidence
> that
> > > > > proves
> > > > > > > > this
> > > > > > > > > > > > >>>>> would harm performance, and the amount of
> > > inaccuracy
> > > > > > using
> > > > > > > > > > > > >>>>> the cached system time could incur is actually
> > > > > > substantial.
> > > > > > > > > But,
> > > > > > > > > > > > >>>>> if you want to just "not mention this" in the
> KIP,
> > > we
> > > > > can
> > > > > > > > defer
> > > > > > > > > > to
> > > > > > > > > > > > >>>>> the actual PR discussion, at which time we're
> in a
> > > > > better
> > > > > > > > > > position
> > > > > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Along the lines of the measurement accuracy
> > > discussion,
> > > > > > one
> > > > > > > > > > > > >>>>> minor thought I had is that maybe we should
> > > consider
> > > > > > > > measuring
> > > > > > > > > > > > >>>>> the task staleness metric at the sink, rather
> than
> > > the
> > > > > > > > source,
> > > > > > > > > so
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >>>>> it includes the processing latency of the task
> > > itself,
> > > > > > not
> > > > > > > > just
> > > > > > > > > > the
> > > > > > > > > > > > >>>> latency
> > > > > > > > > > > > >>>>> of everything up to, but not including, the
> task
> > > (which
> > > > > > > seems
> > > > > > > > > > > > >> confusing
> > > > > > > > > > > > >>>>> for users). I guess this could also be an
> > > > > implementation
> > > > > > > > > detail,
> > > > > > > > > > > > >>> though.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > > > > >>>>> -John
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie
> Blee-Goldman
> > > > > > wrote:
> > > > > > > > > > > > >>>>>> Hey all,
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> After discussing with Bruno I'd like to
> propose a
> > > > > small
> > > > > > > > > > amendment,
> > > > > > > > > > > > >>>>>> which is to record the processor-node-level
> > > metrics
> > > > > only
> > > > > > > for
> > > > > > > > > > > > >>> *stateful*
> > > > > > > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > > > > > > "processor-node-level"
> > > > > > > > > > > > >>>>>> metric and not a "state-store-level" metric
> as the
> > > > > > > staleness
> > > > > > > > > is
> > > > > > > > > > > > >> still
> > > > > > > > > > > > >>>>>> a property of the node rather than of the
> state
> > > > > itself.
> > > > > > > > > > However, it
> > > > > > > > > > > > >>>> seems
> > > > > > > > > > > > >>>>>> that this information is primarily useful for
> > > stateful
> > > > > > > > > operators
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >>>> might
> > > > > > > > > > > > >>>>>> be exposing state via IQ or otherwise
> dependent
> > > on the
> > > > > > > > record
> > > > > > > > > > time
> > > > > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> It's worth calling out that recent performance
> > > > > > > improvements
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > >>>> metrics
> > > > > > > > > > > > >>>>>> framework mean that we no longer fetch the
> system
> > > time
> > > > > > at
> > > > > > > > the
> > > > > > > > > > > > >> operator
> > > > > > > > > > > > >>>>>> level, but only once per task. In other words
> the
> > > > > system
> > > > > > > > time
> > > > > > > > > > is not
> > > > > > > > > > > > >>>> updated
> > > > > > > > > > > > >>>>>> between each process as a record flows
> through the
> > > > > > > > > subtopology,
> > > > > > > > > > so
> > > > > > > > > > > > >>>>>> debugging the processor-level latency via the
> > > > > > stateleness
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > > > > operator-level
> > > > > > > > > metrics
> > > > > > > > > > are
> > > > > > > > > > > > >>> not
> > > > > > > > > > > > >>>>>> *useful* relative to the task-level metric.
> > > Upstream
> > > > > > > caching
> > > > > > > > > > and/or
> > > > > > > > > > > > >>>>>> suppression
> > > > > > > > > > > > >>>>>> can still cause a record's staleness at some
> > > > > downstream
> > > > > > > > > stateful
> > > > > > > > > > > > >>>> operator
> > > > > > > > > > > > >>>>>> to deviate from the task-level staleness
> > > (recorded at
> > > > > > the
> > > > > > > > > source
> > > > > > > > > > > > >>> node).
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> Please let me know if you have any concerns
> about
> > > this
> > > > > > > > change.
> > > > > > > > > > The
> > > > > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna
> <
> > > > > > > > > > bruno@confluent.io>
> > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > > > > >>>>>>> I think we do not need it now. If we need
> it, we
> > > can
> > > > > > add
> > > > > > > > it.
> > > > > > > > > > Here
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>>>>> old truism applies: Adding is always easier
> than
> > > > > > > removing.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > > > > >>>>>>> I think it is good to have the staleness
> metrics
> > > also
> > > > > > on
> > > > > > > > > > > > >>>>>>> processor-node-level. If we do not want to
> record
> > > > > them
> > > > > > on
> > > > > > > > all
> > > > > > > > > > > > >>>>>>> processor nodes, you could restrict the
> > > recording to
> > > > > > > > stateful
> > > > > > > > > > > > >>>>>>> processor-nodes, since those are the ones
> that
> > > would
> > > > > > > > benefit
> > > > > > > > > > most
> > > > > > > > > > > > >>> from
> > > > > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> Best,
> > > > > > > > > > > > >>>>>>> Bruno
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie
> > > Blee-Goldman <
> > > > > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to align
> > > with the
> > > > > > > > current
> > > > > > > > > > > > >>> metrics.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Is it better to conform than to be right?
> > > History
> > > > > has
> > > > > > a
> > > > > > > > lot
> > > > > > > > > > to say
> > > > > > > > > > > > >>> on
> > > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > > >>>>>>>> matter
> > > > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies to
> the
> > > fine
> > > > > > > > details
> > > > > > > > > of
> > > > > > > > > > > > >>> metrics
> > > > > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> More seriously, I figured if people are
> looking
> > > at
> > > > > > this
> > > > > > > > > metric
> > > > > > > > > > > > >>> they're
> > > > > > > > > > > > >>>>>>>> likely to
> > > > > > > > > > > > >>>>>>>> be looking at all the others. Then naming
> this
> > > one
> > > > > > > "-mean"
> > > > > > > > > > would
> > > > > > > > > > > > >>>> probably
> > > > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg"
> suffix in
> > > the
> > > > > > > other
> > > > > > > > > > metrics
> > > > > > > > > > > > >>> has
> > > > > > > > > > > > >>>> a
> > > > > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> As for the percentiles, I actually like p99
> (and
> > > > > p75)
> > > > > > > > > better.
> > > > > > > > > > I'll
> > > > > > > > > > > > >>>> swap
> > > > > > > > > > > > >>>>>>>> that out
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John
> Roesler <
> > > > > > > > > > vvcephei@apache.org
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is
> there a
> > > > > > reason
> > > > > > > to
> > > > > > > > > > choose
> > > > > > > > > > > > >>>> "avg"
> > > > > > > > > > > > >>>>>>>>> instead
> > > > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and
> I
> > > might
> > > > > be
> > > > > > > > > > > > >> oversensitive
> > > > > > > > > > > > >>>>>>> because
> > > > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but it
> > > strikes
> > > > > me
> > > > > > > > that
> > > > > > > > > > "avg"
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >>>>>>>>> actually
> > > > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of
> > > statistics,
> > > > > > > > whereas
> > > > > > > > > > "mean"
> > > > > > > > > > > > >>> is
> > > > > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but
> none
> > > with
> > > > > > > > "mean";
> > > > > > > > > > was
> > > > > > > > > > > > >>> that
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Regarding the names of the percentile, I
> > > actually
> > > > > > > > couldn't
> > > > > > > > > > find
> > > > > > > > > > > > >>> _any_
> > > > > > > > > > > > >>>>>>> other
> > > > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there a
> > > reason to
> > > > > > > choose
> > > > > > > > > > "99th"
> > > > > > > > > > > > >> as
> > > > > > > > > > > > >>>>>>> opposed
> > > > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> > > > > > criticism,
> > > > > > > > I'm
> > > > > > > > > > just
> > > > > > > > > > > > >>>>>>> primarily
> > > > > > > > > > > > >>>>>>>>> asking
> > > > > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > > > >>>>>>>>> -John
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie
> > > Blee-Goldman
> > > > > > > > wrote:
> > > > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min
> > > metric
> > > > > for
> > > > > > > the
> > > > > > > > > > sake of
> > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean
> (or
> > > 50th
> > > > > > > > > > percentile)
> > > > > > > > > > > > >>> but I
> > > > > > > > > > > > >>>>>>> can
> > > > > > > > > > > > >>>>>>>>> see
> > > > > > > > > > > > >>>>>>>>>> how users might expect it and find it a
> bit
> > > > > > > disorienting
> > > > > > > > > > not to
> > > > > > > > > > > > >>>>>>> have. So
> > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > > > > percentile)*
> > > > > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > > > > percentile)*
> > > > > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John
> Roesler <
> > > > > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to
> > > embarrass
> > > > > > > > myself.
> > > > > > > > > I
> > > > > > > > > > > > >> guess
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>> mean
> > > > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the
> > > median, but
> > > > > > > > neither
> > > > > > > > > > are
> > > > > > > > > > > > >> as
> > > > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles
> (99th
> > > and
> > > > > > max).
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs,
> but
> > > it
> > > > > > does
> > > > > > > > > round
> > > > > > > > > > out
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>>>>>>> mental
> > > > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always
> > > graphed
> > > > > > min
> > > > > > > > > along
> > > > > > > > > > with
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > > >>>>>>>>>>> metrics to help me understand how fast
> the
> > > system
> > > > > > can
> > > > > > > > be,
> > > > > > > > > > which
> > > > > > > > > > > > >>>>>>> helps
> > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a
> > > relatively
> > > > > > > > > inexpensive
> > > > > > > > > > > > >> metric
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just
> throw
> > > it in.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> > > > > Blee-Goldman
> > > > > > > > > wrote:
> > > > > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end
> > > latency
> > > > > > *up*
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if
> that
> > > drives
> > > > > > the
> > > > > > > > > point
> > > > > > > > > > > > >> home
> > > > > > > > > > > > >>>>>>>>> better.
> > > > > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is
> > > received by
> > > > > > that
> > > > > > > > > > task" --
> > > > > > > > > > > > >>>>>>> will
> > > > > > > > > > > > >>>>>>>>> update
> > > > > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where
> the
> > > min
> > > > > > would
> > > > > > > > > > actually
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't
> > > feel
> > > > > > > strongly
> > > > > > > > > > that we
> > > > > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the start.
> Can
> > > you
> > > > > > > > > enlighten
> > > > > > > > > > me
> > > > > > > > > > > > >>>>>>> with an
> > > > > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the
> > > overhead
> > > > > of
> > > > > > > > > adding
> > > > > > > > > > > > >>>>>>> multiple
> > > > > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate
> > > what
> > > > > kind
> > > > > > > of
> > > > > > > > > > > > >>> performance
> > > > > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the
> 50th
> > > > > > percentile
> > > > > > > > > > would be
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>>>>>>> median
> > > > > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean
> > > *and*
> > > > > the
> > > > > > > 50th
> > > > > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John
> > > Roesler <
> > > > > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP,
> Sophie,
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so
> > > far:
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter
> whether
> > > we
> > > > > > report
> > > > > > > > > > "spans"
> > > > > > > > > > > > >> or
> > > > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice,
> > > some of
> > > > > > the
> > > > > > > > > spans
> > > > > > > > > > are
> > > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like time
> > > spent
> > > > > > > waiting
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > > > > >>>>>>>>>>>>> time from the event happening to the
> ETL
> > > > > producer
> > > > > > > > > > choosing to
> > > > > > > > > > > > >>>>>>> send
> > > > > > > > > > > > >>>>>>>>> it,
> > > > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers,
> > > etc.,
> > > > > etc.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically
> easier to
> > > > > > compute
> > > > > > > > > spans
> > > > > > > > > > by
> > > > > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e
> > > > > latencies
> > > > > > > by
> > > > > > > > > > adding
> > > > > > > > > > > > >>>>>>> spans.
> > > > > > > > > > > > >>>>>>>>> You
> > > > > > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> > > > > computation
> > > > > > > from
> > > > > > > > > e2e
> > > > > > > > > > > > >>>>>>> always
> > > > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers,
> whereas
> > > > > > computing
> > > > > > > > e2e
> > > > > > > > > > > > >> latency
> > > > > > > > > > > > >>>>>>>>> from
> > > > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans
> > > leading
> > > > > up
> > > > > > to
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > > >> you
> > > > > > > > > > > > >>>>>>>>> care
> > > > > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to
> have
> > > > > spans
> > > > > > > when
> > > > > > > > > > they
> > > > > > > > > > > > >> are
> > > > > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency
> is a
> > > more
> > > > > > > > general
> > > > > > > > > > > > >>>>>>> measurement
> > > > > > > > > > > > >>>>>>>>>>>>> that basically every person/application
> > > cares
> > > > > > about
> > > > > > > > and
> > > > > > > > > > > > >> should
> > > > > > > > > > > > >>>>>>> be
> > > > > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide
> more
> > > > > value
> > > > > > to
> > > > > > > > > more
> > > > > > > > > > > > >>>>>>> people if
> > > > > > > > > > > > >>>>>>>>> we
> > > > > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding
> > > "record-staleness" as
> > > > > a
> > > > > > > > name,
> > > > > > > > > I
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >>>>>>> I
> > > > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> > > > > > intuition.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is
> nice,
> > > > > since
> > > > > > > the
> > > > > > > > > > inside
> > > > > > > > > > > > >> of
> > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > >>>>>>>>> task
> > > > > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of
> latency
> > > in
> > > > > some
> > > > > > > > > cases.
> > > > > > > > > > > > >> Plus,
> > > > > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really
> wanted to
> > > > > know
> > > > > > > (for
> > > > > > > > > the
> > > > > > > > > > > > >>>>>>> purposes
> > > > > > > > > > > > >>>>>>>>> of
> > > > > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes source
> > > events
> > > > > to
> > > > > > > > "show
> > > > > > > > > > up" at
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at every
> > > > > processor
> > > > > > > > could
> > > > > > > > > be
> > > > > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at
> the
> > > node
> > > > > > > level.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to
> record
> > > also.
> > > > > > > Even
> > > > > > > > if
> > > > > > > > > > it's
> > > > > > > > > > > > >> a
> > > > > > > > > > > > >>>>>>> poor
> > > > > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of
> people
> > > might
> > > > > > > > expect
> > > > > > > > > > to see
> > > > > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is
> there
> > > a
> > > > > > reason
> > > > > > > > to
> > > > > > > > > > leave
> > > > > > > > > > > > >> it
> > > > > > > > > > > > >>>>>>>>> off?
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > > > > >>>>>>>>>>>>> I agree we should include late records
> > > (though
> > > > > > not
> > > > > > > > the
> > > > > > > > > > ones
> > > > > > > > > > > > >> we
> > > > > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there
> are
> > > > > > > legitimately
> > > > > > > > > > some
> > > > > > > > > > > > >>>>>>> records
> > > > > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the
> whole
> > > > > point
> > > > > > > of
> > > > > > > > > > these
> > > > > > > > > > > > >>>>>>> metrics.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any
> other
> > > > > > feedback,
> > > > > > > > > other
> > > > > > > > > > > > >> than
> > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58,
> Guozhang
> > > Wang
> > > > > > > wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick
> > > > > thoughts:
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes
> both
> > > the
> > > > > > > > processing
> > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in
> > > intermediate
> > > > > > > > topics.
> > > > > > > > > I
> > > > > > > > > > have
> > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above
> that the
> > > > > > latency
> > > > > > > > > > metric of
> > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > >>>>>>>>> task A
> > > > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > > > > > sub-topology
> > > > > > > > > up-to
> > > > > > > > > > but
> > > > > > > > > > > > >>>>>>> not
> > > > > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit
> weird.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the
> name
> > > > > > "latency"
> > > > > > > > > > itself,
> > > > > > > > > > > > >>>>>>> since
> > > > > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics
> > > already
> > > > > > > which
> > > > > > > > > are
> > > > > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a record /
> etc,
> > > > > while
> > > > > > > > here
> > > > > > > > > > we are
> > > > > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the record
> > > > > timestamp.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > > > > > "record-staleness"
> > > > > > > > > > > > >>>>>>> (note we
> > > > > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in
> which
> > > > > case
> > > > > > > > > > recording at
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>> system-time before we start
> processing the
> > > > > > record
> > > > > > > > > sounds
> > > > > > > > > > > > >> more
> > > > > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering
> if the
> > > > > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we
> > > already
> > > > > > > have a
> > > > > > > > > > > > >>>>>>> task-level
> > > > > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific
> > > node's
> > > > > e2e
> > > > > > > > > > latency is
> > > > > > > > > > > > >>>>>>>>> similar
> > > > > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level
> > > processing
> > > > > > > > > latency.
> > > > > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness
> > > metric is
> > > > > > > > > > sufficient.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM
> Sophie
> > > > > > > > Blee-Goldman <
> > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a
> particularly
> > > > > > useful
> > > > > > > > > gauge
> > > > > > > > > > for
> > > > > > > > > > > > >>>>>>> this
> > > > > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at
> putting an
> > > > > > *upper
> > > > > > > > > > *bound on
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose
> > > percentiles
> > > > > > > that
> > > > > > > > > > would
> > > > > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will
> > > experience,
> > > > > > and
> > > > > > > > > what
> > > > > > > > > > > > >>>>>>> *close
> > > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these
> specific
> > > > > > numbers
> > > > > > > > and
> > > > > > > > > > > > >>>>>>> could be
> > > > > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in
> hearing
> > > > > from
> > > > > > > > users
> > > > > > > > > on
> > > > > > > > > > > > >>>>>>> this.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> > > > > "hop-to-hop
> > > > > > > > > > latency" in
> > > > > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> > > > > subtracting
> > > > > > > the
> > > > > > > > > > > > >>>>>>> previous
> > > > > > > > > > > > >>>>>>>>> node's
> > > > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could
> do
> > > it
> > > > > > either
> > > > > > > > way
> > > > > > > > > > since
> > > > > > > > > > > > >>>>>>>>> you can
> > > > > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I
> think
> > > the
> > > > > > > > > end-to-end
> > > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is
> not
> > > to
> > > > > > debug
> > > > > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > > > > >>>>>>>>> in the
> > > > > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it
> > > takes
> > > > > > > arecord
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this
> might
> > > be
> > > > > > useful
> > > > > > > > for
> > > > > > > > > > users
> > > > > > > > > > > > >>>>>>>>> who are
> > > > > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just
> > > produced
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > >>>>>>> included
> > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect
> --
> > > but
> > > > > > > maybe I
> > > > > > > > > > didn't
> > > > > > > > > > > > >>>>>>> make
> > > > > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address
> this
> > > in
> > > > > the
> > > > > > > KIP.
> > > > > > > > > The
> > > > > > > > > > > > >>>>>>> short
> > > > > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I
> added a
> > > > > > paragraph
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>> end
> > > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the
> reasoning
> > > > > here,
> > > > > > > > please
> > > > > > > > > > let
> > > > > > > > > > > > >>>>>>> me
> > > > > > > > > > > > >>>>>>>>> know
> > > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the
> > > existing
> > > > > > > metric
> > > > > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > > > > Node#process
> > > > > > > > method
> > > > > > > > > > to
> > > > > > > > > > > > >>>>>>> run
> > > > > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative to
> the
> > > > > event
> > > > > > > > > > timestamp.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision
> there is
> > > > > pretty
> > > > > > > > > > confusing
> > > > > > > > > > > > >>>>>>> so
> > > > > > > > > > > > >>>>>>>>> I've
> > > > > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to
> > > "end-to-end-latency"
> > > > > > > which I
> > > > > > > > > > feel
> > > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM
> Boyang
> > > Chen
> > > > > <
> > > > > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting
> the
> > > E2E
> > > > > > > latency
> > > > > > > > > is
> > > > > > > > > > > > >>>>>>>>> important
> > > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> > > > > > application.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of
> > > picking
> > > > > > 75,
> > > > > > > > 99
> > > > > > > > > > and
> > > > > > > > > > > > >>>>>>> max
> > > > > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use
> 50, 90
> > > > > > > > percentile
> > > > > > > > > as
> > > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being
> computed is
> > > > > > > > cumulative,
> > > > > > > > > > I.E
> > > > > > > > > > > > >>>>>>> if a
> > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) =
> > > T(B->C) +
> > > > > > P(B)
> > > > > > > =
> > > > > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the
> > > captured
> > > > > > > > > latency,
> > > > > > > > > > > > >>>>>>> and T()
> > > > > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records
> > > between
> > > > > > two
> > > > > > > > > nodes,
> > > > > > > > > > > > >>>>>>>>> including
> > > > > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe
> > > having
> > > > > > > T(B->C)
> > > > > > > > > and
> > > > > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop
> latency",
> > > > > > > otherwise
> > > > > > > > > if
> > > > > > > > > > > > >>>>>>> there
> > > > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are
> > > affected in
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > >>>>>>> time.
> > > > > > > > > > > > >>>>>>>>> In the
> > > > > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is
> meaningful
> > > only
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > > >>>>>>> record
> > > > > > > > > > > > >>>>>>>>> exits
> > > > > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time
> this
> > > > > record
> > > > > > > > spent
> > > > > > > > > > > > >>>>>>> inside
> > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate
> > > treatment
> > > > > for
> > > > > > > > sink
> > > > > > > > > > > > >>>>>>> nodes and
> > > > > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only
> count
> > > the
> > > > > time
> > > > > > > > > > receiving
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a
> solution
> > > here,
> > > > > > > just
> > > > > > > > > > want to
> > > > > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is
> reasonable.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late
> > > arrival
> > > > > > > records
> > > > > > > > > as
> > > > > > > > > > > > >>>>>>> well,
> > > > > > > > > > > > >>>>>>>>> they
> > > > > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs
> when the
> > > > > > > > > out-of-order
> > > > > > > > > > > > >>>>>>>>> records are
> > > > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records.
> > > Should we
> > > > > > > also
> > > > > > > > > > supply
> > > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user
> should
> > > just
> > > > > > take
> > > > > > > > > care
> > > > > > > > > > of
> > > > > > > > > > > > >>>>>>> it
> > > > > > > > > > > > >>>>>>>>> by
> > > > > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we
> > > haven't
> > > > > > > > discussed
> > > > > > > > > > its
> > > > > > > > > > > > >>>>>>>>> relation
> > > > > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics,
> > > could
> > > > > you
> > > > > > > add
> > > > > > > > > > some
> > > > > > > > > > > > >>>>>>>>> context
> > > > > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use
> > > which`
> > > > > > > > tutorial
> > > > > > > > > > for
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM
> Sophie
> > > > > > > > Blee-Goldman
> > > > > > > > > <
> > > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on
> > > KIP-613
> > > > > > > which
> > > > > > > > > aims
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>>> add
> > > > > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please
> > > take a
> > > > > > look:
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Bruno Cadonna <br...@confluent.io>.
Hi Sophie,

Just a question for my understanding. When you say " t_A be the time
when it is finished being processed by the aggregator node", this
means before caching (or putting into the state store if caching is
disabled), right?

Best,
Bruno

On Sat, May 16, 2020 at 6:57 PM Guozhang Wang <wa...@gmail.com> wrote:
>
> We do have a big JIRA for making sensor recording to be sufficient. But it
> is a bit hard to try to reduce the num.system clock calls in general (but
> admittedly, in nowadays OS they are supposed to be very cheap as well),
> I've just recently merged a small PR trying to make less
> SystemTime.currentMilliseconds calls whenever possible, but it is hard to
> enforce that for all callers.
>
> Guozhang
>
> On Sat, May 16, 2020 at 9:46 AM John Roesler <vv...@apache.org> wrote:
>
> > FWIW, I’d be fine with TRACE, if it makes the processor-level metrics more
> > palatable to everyone.
> >
> > Ultimately, I think this is just kicking the can down the road, and we
> > should really just refactor our sensors/metrics to be lower overhead. But
> > that’s too much work to consider right before the release so for now, we
> > can kick that can just a little farther.
> >
> > Thanks for proposing a solution, Guozhang!
> >
> > -John
> >
> > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> > > Cool, that's good points -- we should not let our current metrics
> > proposal
> > > to depend on future roadmaps, since they can still be removed later if
> > ever
> > > becoming not valuable after all.
> > >
> > > On the other hand, what do you think about introducing TRACE level
> > metrics
> > > recording and make the the processor-node level metrics on TRACE?
> > >
> > > Guozhang
> > >
> > > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <sophie@confluent.io
> > >
> > > wrote:
> > >
> > > > Yeah, I had proposed to replace the "source + sink" task-level metric
> > > > with "sink only" task-level metrics plus the stateful node-level ones.
> > > >
> > > > I completely agree that this proposal should and would look different
> > > > once we complete those two proposals, ie decouple caching and
> > > > consolidate suppression. But as it is right now, caching and
> > suppression
> > > > cause the actual time-of-processing of records to vary greatly
> > throughout
> > > > a subtopology such that the delta between a source node and an
> > intermediate
> > > > one is potentially quite large.
> > > >
> > > > For this exact reason I would argue that this metric is still valuable
> > at
> > > > the node-level
> > > >  even if the "current time" is actually only updated once for a task.
> > > > Obviously this
> > > > would not convey any information about the actual processing latency,
> > but
> > > > as you
> > > > stated, this is likely to be a small delta on the scale of
> > milliseconds.
> > > > But the latency
> > > > due to a suppression can be on the order of minutes, hours, or even
> > days!
> > > > If
> > > > there is more than one suppression or caching in a subtopology, the
> > source
> > > > and sink
> > > > e2e latency could be wildly misrepresentative of the e2e latency of a
> > node
> > > > in the middle.
> > > >
> > > > Anyways, I removed this last point from the current KIP document
> > because I
> > > > felt
> > > > it was ultimately an implementation detail. But the larger point I
> > would
> > > > argue is that:
> > > > if we are truly worried about the performance hit of fetching the
> > system
> > > > time at every
> > > > node, we should just not fetch the system time at every node instead of
> > > > dropping the
> > > > metric altogether. It may be slightly off but only on the order of ms
> > and
> > > > not on the order
> > > > of hours, and it's still more useful than no metric at all
> > > >
> > > >
> > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Sophie,
> > > > >
> > > > > I think your motivation makes sense, in fact before KIP-444 we used
> > to
> > > > have
> > > > > processor-node-level process latency as DEBUG which would
> > holistically
> > > > > address the motivation but we removed it since it is indeed quite
> > > > expensive
> > > > > (I do not have concrete numbers at hand, but basically you'd have to
> > call
> > > > > system-wall-clock time a couple of time for each record, each
> > processor
> > > > > node), and in the long run I feel such scenarios would be less
> > common if
> > > > we
> > > > > 1) decouple caching from emitting and 2) always put suppression right
> > > > > before sink only, and that's also why I've proposed to remove them in
> > > > > KIP-444.
> > > > >
> > > > > I missed an update in your wiki that right now you've updated the
> > > > > task-level metric to be recorded at the sink node, and thought it is
> > > > still
> > > > > at the source node, but assuming it is still the case, then
> > practically
> > > > any
> > > > > state store's staleness of the task could roughly be measured as that
> > > > > task-level staleness plus a small delta (usually tens of mills), and
> > the
> > > > > down stream task B's staleness - up stream task A's staleness would
> > > > capture
> > > > > the suppression effect at the end of A plus the intermediate topic
> > > > > producing -> consuming latency, which usually would not require to be
> > > > > distinguished.
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <
> > sophie@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Guozhang,
> > > > > >
> > > > > > Thanks for the response. I meant to ask this earlier and forgot,
> > but do
> > > > > you
> > > > > > have any data or benchmarking results on hand for the performance
> > hit
> > > > we
> > > > > > took due to the metrics? Or maybe better yet, on the performance we
> > > > > gained
> > > > > > back by dropping some metrics? I think it would help us to better
> > > > > > understand
> > > > > > the tradeoffs with some concrete numbers to compare.
> > > > > >
> > > > > > I guess I'm inclined to want to include these metrics at the
> > operator
> > > > > level
> > > > > > on
> > > > > > DEBUG since, in my mind, that's exactly what DEBUG is for:
> > performance-
> > > > > > focused users will presumably run at INFO level and even those who
> > do
> > > > > > use DEBUG may only do so temporarily, when they actually have
> > something
> > > > > > to debug.
> > > > > >
> > > > > > Of course, I'm not running a Streams app myself so all this is just
> > > > > > conjecture.
> > > > > > But it seems reasonable to interpret the two metrics levels as
> > "minimal
> > > > > > useful
> > > > > > information" and "all information that provides insight into the
> > > > system"
> > > > > >
> > > > > > If we can agree there, the question becomes "does this provide any
> > > > > > additional
> > > > > > insight?" I would still argue that it does. To give a concrete
> > example,
> > > > > > let's say
> > > > > > the user has a basic
> > > > > >
> > > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > > > >
> > > > > > subtopology, the application performing IQ on the transformer's
> > state.
> > > > > The
> > > > > > user
> > > > > > wants to know how long it's taking records to be reflected in the
> > > > results
> > > > > > of a query.
> > > > > > With only task-level metrics, they can only guess based on the e2e
> > > > > latency
> > > > > > of the
> > > > > > previous task and of this one. But neither of those is a good
> > > > > > representation of the
> > > > > > time it will take a record to be processed by the transformer: the
> > > > > previous
> > > > > > task's
> > > > > > latency will not include the time for the record to go through the
> > > > > > repartitioning,
> > > > > > and the current task's latency is measured after a suppression. The
> > > > > > suppression
> > > > > > itself will introduce significant latency and doesn't provide an
> > > > accurate
> > > > > > estimate.
> > > > > > Caching would also do what it does best, messing with our
> > understanding
> > > > > of
> > > > > > time :)
> > > > > >
> > > > > > It also means you can't measure the consumption latency, so there
> > would
> > > > > be
> > > > > > no way to tell if the e2e latency is so high because the very first
> > > > > > subtopology is
> > > > > > extremely slow, or the record was severely delayed in reaching the
> > > > input
> > > > > > topic.
> > > > > > I guess my earlier proposal to include both source and sink metrics
> > > > could
> > > > > > address this issue, but it seems to solve the problem more
> > holistically
> > > > > --
> > > > > > and for
> > > > > > more complex subtopologies -- by including metrics at the processor
> > > > level
> > > > > >
> > > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hey folks,
> > > > > > >
> > > > > > > I want to make a final fight on processor-node-level metrics :)
> > Post
> > > > > > > KIP-444 we've actually removed a lot of node-level metrics since
> > it
> > > > is
> > > > > > too
> > > > > > > expensive to record at that level and their values are not proven
> > > > worth
> > > > > > it.
> > > > > > > Again I'd use the "we have to either enable all or none of DEBUG
> > > > > metrics"
> > > > > > > card, e.g. if we want to look into per-task process-latency
> > which is
> > > > > > DEBUG,
> > > > > > > we'd have to record all process-node / state-store / cache level
> > > > > metrics.
> > > > > > > On the other hand, within a task we usually do not have a lot of
> > > > > stateful
> > > > > > > operator nodes and if we want to find if certain stateful nodes
> > are
> > > > the
> > > > > > > bottleneck of process latency, that is usually discoverable from
> > the
> > > > > > > state-store level read / write latencies already. So I'd imagine
> > that
> > > > > > > suppose there's a sub-topology with multiple state stores in it,
> > if
> > > > we
> > > > > > > found the task-level process latency is high, we can still tell
> > which
> > > > > > state
> > > > > > > stores within it is the main bottleneck from the state store
> > read /
> > > > > write
> > > > > > > latency (I'm assuming the IO latency is always dominant compared
> > with
> > > > > > > others).
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > > > sophie@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > @Matthias
> > > > > > > > > And that's what he says about the 95th percentile! Imagine
> > what
> > > > he
> > > > > > > would
> > > > > > > > say about the 50th :P
> > > > > > > >
> > > > > > > > I should have kept watching another 10 minutes. He gets around
> > to
> > > > > > > covering
> > > > > > > > the 50th, and let's
> > > > > > > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <
> > vvcephei@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks, Sophie!
> > > > > > > > >
> > > > > > > > > I think this makes perfect sense. It will be much more
> > intuitive
> > > > to
> > > > > > use
> > > > > > > > > the metric for the stated motivation this way.  I’d be in
> > favor
> > > > of
> > > > > > the
> > > > > > > > > proposal after this update.
> > > > > > > > >
> > > > > > > > > Thanks again for taking this on,
> > > > > > > > > -John
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to clarify/modify one aspect of this KIP, which
> > is to
> > > > > > record
> > > > > > > > the
> > > > > > > > > > staleness
> > > > > > > > > > at the *completion* of the record's processing by the
> > > > > operator/task
> > > > > > > in
> > > > > > > > > > question,
> > > > > > > > > > rather than on the intake. The task-level metrics will be
> > > > > recorded
> > > > > > at
> > > > > > > > the
> > > > > > > > > > sink
> > > > > > > > > > node instead of at the source, and the operator-level
> > metrics
> > > > > will
> > > > > > be
> > > > > > > > > > recorded
> > > > > > > > > > at the end of the operation.
> > > > > > > > > >
> > > > > > > > > > The stated purpose and intended usefulness of this KIP is
> > to
> > > > give
> > > > > > > > users a
> > > > > > > > > > way
> > > > > > > > > > to gauge roughly how long it takes for a record to be
> > reflected
> > > > > in
> > > > > > > the
> > > > > > > > > > "results",
> > > > > > > > > > whether these results are being read from an output topic
> > or
> > > > > > through
> > > > > > > > IQ.
> > > > > > > > > To
> > > > > > > > > > take the IQ example, the results of a record are obviously
> > not
> > > > > > > visible
> > > > > > > > > until
> > > > > > > > > > *after* that node has finished processing it. The
> > staleness,
> > > > > while
> > > > > > > > still
> > > > > > > > > > potentially
> > > > > > > > > > useful as it can impact the *way* a record is processed in
> > a
> > > > > > stateful
> > > > > > > > and
> > > > > > > > > > time-
> > > > > > > > > > dependent operator, is not part of the problem this KIP
> > > > > > specifically
> > > > > > > > set
> > > > > > > > > out
> > > > > > > > > > to solve.
> > > > > > > > > >
> > > > > > > > > > In light of this I think it's appropriate to revert the
> > name
> > > > > change
> > > > > > > > back
> > > > > > > > > to
> > > > > > > > > > include
> > > > > > > > > > latency, since "staleness" as described above only makes
> > sense
> > > > > when
> > > > > > > > > > measuring
> > > > > > > > > > relative to the arrival of a record at a task/node. I'd
> > propose
> > > > > to
> > > > > > > save
> > > > > > > > > the
> > > > > > > > > > term
> > > > > > > > > > "staleness" for that particular meaning, and adopt
> > Matthias's
> > > > > > > > suggestion
> > > > > > > > > of
> > > > > > > > > > "record-e2e-latency" for this.
> > > > > > > > > >
> > > > > > > > > > Thanks for hanging in there all. Please let me know if you
> > have
> > > > > any
> > > > > > > > > > concerns about
> > > > > > > > > > this change!
> > > > > > > > > >
> > > > > > > > > > Sophie
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > > > mjsax@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I am also happy with max/min/99/90. And I buy your naming
> > > > > > argument
> > > > > > > > > about
> > > > > > > > > > > staleness vs latency.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -Matthias
> > > > > > > > > > >
> > > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > > > Hey Sophie,
> > > > > > > > > > > >
> > > > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > > > > > > sophie@confluent.io>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> @Matthias
> > > > > > > > > > > >> Regarding tracking the 50th percentile, I'll refer
> > you to
> > > > > the
> > > > > > > 4:53
> > > > > > > > > mark
> > > > > > > > > > > of
> > > > > > > > > > > >> the video
> > > > > > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > > > >>
> > > > > > > > > > > >> And that's what he says about the 95th percentile!
> > Imagine
> > > > > > what
> > > > > > > he
> > > > > > > > > would
> > > > > > > > > > > >> say about
> > > > > > > > > > > >> the 50th :P
> > > > > > > > > > > >>
> > > > > > > > > > > >> But seriously, since we can't seem to agree that the
> > mean
> > > > or
> > > > > > > 50th
> > > > > > > > > > > >> percentile is actually
> > > > > > > > > > > >> useful I'm inclined to resurrect my original proposal,
> > > > > > neither.
> > > > > > > > But
> > > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > >> that's a good
> > > > > > > > > > > >> argument against the 75th, which I admittedly chose
> > > > somewhat
> > > > > > > > > > > arbitrarily as
> > > > > > > > > > > >> an
> > > > > > > > > > > >> intermediate between the 50th and the higher
> > percentiles.
> > > > > How
> > > > > > > > about:
> > > > > > > > > > > >>
> > > > > > > > > > > >> -max
> > > > > > > > > > > >> -p99
> > > > > > > > > > > >> -p90
> > > > > > > > > > > >> -min
> > > > > > > > > > > >>
> > > > > > > > > > > >> with p50/mean still up for debate if anyone feels
> > strongly
> > > > > for
> > > > > > > > > either of
> > > > > > > > > > > >> them.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regarding the name, I've already flip-flopped on this
> > so
> > > > I'm
> > > > > > > > > definitely
> > > > > > > > > > > >> still open to
> > > > > > > > > > > >> further arguments. But the reason for changing it from
> > > > > > > > > > > end-to-end-latency
> > > > > > > > > > > >> (which
> > > > > > > > > > > >> is similar to what you propose) is that this metric
> > > > > > technically
> > > > > > > > > reflects
> > > > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > > > >> the record is when it's *received* by the operator,
> > not
> > > > when
> > > > > > > it's
> > > > > > > > > > > processed
> > > > > > > > > > > >> by the operator.
> > > > > > > > > > > >> It seemed like there was the potential for confusion
> > that
> > > > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > > > >> represent the latency from the event creation to the
> > time
> > > > > the
> > > > > > > > > processor
> > > > > > > > > > > is
> > > > > > > > > > > >> done
> > > > > > > > > > > >> processing it.
> > > > > > > > > > > >>
> > > > > > > > > > > >> @John
> > > > > > > > > > > >> I'd rather err on the side of "not-enough" metrics as
> > we
> > > > can
> > > > > > > > always
> > > > > > > > > add
> > > > > > > > > > > >> this to the
> > > > > > > > > > > >> stateless metrics later on. If we decide to measure
> > the
> > > > time
> > > > > > at
> > > > > > > > > every
> > > > > > > > > > > node
> > > > > > > > > > > >> and don't
> > > > > > > > > > > >> find any evidence of a serious performance impact, and
> > > > users
> > > > > > > > > indicate
> > > > > > > > > > > they
> > > > > > > > > > > >> would
> > > > > > > > > > > >> like to see this metric at all nodes, then we can
> > easily
> > > > > start
> > > > > > > > > reporting
> > > > > > > > > > > >> them as well.
> > > > > > > > > > > >> WDYT?
> > > > > > > > > > > >>
> > > > > > > > > > > >> That said, sink nodes seem like a reasonable
> > exception to
> > > > > the
> > > > > > > > rule.
> > > > > > > > > > > >> Obviously users
> > > > > > > > > > > >> should be able to detect the time when the record
> > reaches
> > > > > the
> > > > > > > > output
> > > > > > > > > > > topic
> > > > > > > > > > > >> but that
> > > > > > > > > > > >> still leaves a gap in understanding how long the
> > > > production
> > > > > > > > latency
> > > > > > > > > was.
> > > > > > > > > > > >> This mirrors
> > > > > > > > > > > >> the consumption latency that is exposed by the
> > task-level
> > > > > > > metrics,
> > > > > > > > > which
> > > > > > > > > > > >> are measured
> > > > > > > > > > > >> at the source node. For good symmetry what if we
> > actually
> > > > > > expose
> > > > > > > > > both
> > > > > > > > > > > the
> > > > > > > > > > > >> source
> > > > > > > > > > > >> and sink latency at the task-level? ie report both
> > sets of
> > > > > > > > > statistical
> > > > > > > > > > > >> measurements with
> > > > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > > > >>
> > > > > > > > > > > >> @Bill
> > > > > > > > > > > >> Thanks for the comment about regarding the min! I
> > hadn't
> > > > > > > > considered
> > > > > > > > > that
> > > > > > > > > > > >> and it's
> > > > > > > > > > > >> quite useful to think about how and what is useful
> > from a
> > > > > > users
> > > > > > > > > point of
> > > > > > > > > > > >> view.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regarding your second. point, I'm inclined to leave
> > that
> > > > as
> > > > > an
> > > > > > > > > > > >> implementation detail
> > > > > > > > > > > >> but my take would be that the user should be allowed
> > to
> > > > > > control
> > > > > > > > the
> > > > > > > > > > > record
> > > > > > > > > > > >> timestamp
> > > > > > > > > > > >> used for this with the timestamp extractor. My
> > impression
> > > > is
> > > > > > > that
> > > > > > > > > users
> > > > > > > > > > > may
> > > > > > > > > > > >> often embed
> > > > > > > > > > > >> the actual event time in the payload for whatever
> > reason,
> > > > > and
> > > > > > > this
> > > > > > > > > > > >> represents the "true"
> > > > > > > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > > > bbejeck@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful
> > metric
> > > > to
> > > > > > > add.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Regarding tracking min, I  think it could be
> > valuable for
> > > > > > users
> > > > > > > > to
> > > > > > > > > > > >> discern
> > > > > > > > > > > >>> which part of their topologies are more efficient
> > since
> > > > > this
> > > > > > > is a
> > > > > > > > > > > >>> task-level metric.  I realize everyone seems to be on
> > > > board
> > > > > > > with
> > > > > > > > > > > >> including
> > > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on this
> > topic
> > > > > > should
> > > > > > > > we
> > > > > > > > > > > decide
> > > > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> I do have a question regarding the calculation of
> > > > > staleness.
> > > > > > > > > > > >>> Is there going to be a consideration for timestamp
> > > > > > extractors?
> > > > > > > > > Users
> > > > > > > > > > > >> could
> > > > > > > > > > > >>> prefer to use a timestamp embedded in the payload,
> > and it
> > > > > > could
> > > > > > > > > skew
> > > > > > > > > > > the
> > > > > > > > > > > >>> measurements.
> > > > > > > > > > > >>> I was wondering if we should specify in the KIP if
> > > > setting
> > > > > > the
> > > > > > > > > arrival
> > > > > > > > > > > >> time
> > > > > > > > > > > >>> is always going to come from the record timestamp,
> > or is
> > > > > this
> > > > > > > an
> > > > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > >>> Bill
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > > > > > mjsax@apache.org>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> I think it's not useful to record the avg/mean; it
> > > > > sensitive
> > > > > > > to
> > > > > > > > > > > >>>> outliers. We should rather track the median (50th
> > > > > > percentile).
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Not sure if tracking min is useful, but I am also
> > ok to
> > > > > > track
> > > > > > > > it.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> However, I find it odd to track 75th percentile.
> > > > Standard
> > > > > > > > measures
> > > > > > > > > > > >> would
> > > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need both, so
> > maybe
> > > > > > > picking
> > > > > > > > > 90th
> > > > > > > > > > > >>>> might be more useful?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> About the name: "staleness" wound really odd, and if
> > > > fact
> > > > > > the
> > > > > > > > > metric
> > > > > > > > > > > >>>> does capture "latency" so we should call it
> > "latency". I
> > > > > > > > > understand
> > > > > > > > > > > the
> > > > > > > > > > > >>>> issue that we already have a latency metric. So
> > maybe we
> > > > > > could
> > > > > > > > > call it
> > > > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> While I agree that we should include out-or-order
> > data
> > > > > (the
> > > > > > > KIP
> > > > > > > > > should
> > > > > > > > > > > >>>> talk about `out-of-order` data, not `late` data;
> > data is
> > > > > > only
> > > > > > > > > `late`
> > > > > > > > > > > if
> > > > > > > > > > > >>>> it's out-of-order and if it's dropped), I don't
> > really
> > > > > > > > understand
> > > > > > > > > why
> > > > > > > > > > > >>>> the new metric would help to configure grace period
> > or
> > > > > > > retention
> > > > > > > > > time?
> > > > > > > > > > > >>>> As you mention in the KIP, both are define as max
> > > > > difference
> > > > > > > of
> > > > > > > > > > > >>>> `event-time - stream-time` and thus the new metric
> > that
> > > > > > takes
> > > > > > > > > > > >>>> system-/wallclock-time into account does not seem to
> > > > help
> > > > > at
> > > > > > > > all.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> > > > > > Latency"
> > > > > > > by
> > > > > > > > > Gil
> > > > > > > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> -Matthias
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> It seems like there would still be plenty of use
> > cases
> > > > > for
> > > > > > > > > recording
> > > > > > > > > > > >>>>> this metric at all processors and not just stateful
> > > > ones,
> > > > > > but
> > > > > > > > I'm
> > > > > > > > > > > >> happy
> > > > > > > > > > > >>>>> to suspend my arguments for now. Since you're
> > proposing
> > > > > to
> > > > > > > keep
> > > > > > > > > > > >>>>> them at the processor-node level, it will be
> > seamless
> > > > > later
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > >>>>> in the stateless processors if we want. As a wise
> > man
> > > > > once
> > > > > > > > said,
> > > > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Regarding the time measurement, it's an
> > implementation
> > > > > > detail
> > > > > > > > > > > >>>>> we don't need to consider in the KIP.
> > Nevertheless, I'd
> > > > > > > greatly
> > > > > > > > > > > >>>>> prefer to measure the system time again when
> > recording
> > > > > the
> > > > > > > > > > > >>>>> metric. I don't think we've seen any evidence that
> > > > proves
> > > > > > > this
> > > > > > > > > > > >>>>> would harm performance, and the amount of
> > inaccuracy
> > > > > using
> > > > > > > > > > > >>>>> the cached system time could incur is actually
> > > > > substantial.
> > > > > > > > But,
> > > > > > > > > > > >>>>> if you want to just "not mention this" in the KIP,
> > we
> > > > can
> > > > > > > defer
> > > > > > > > > to
> > > > > > > > > > > >>>>> the actual PR discussion, at which time we're in a
> > > > better
> > > > > > > > > position
> > > > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Along the lines of the measurement accuracy
> > discussion,
> > > > > one
> > > > > > > > > > > >>>>> minor thought I had is that maybe we should
> > consider
> > > > > > > measuring
> > > > > > > > > > > >>>>> the task staleness metric at the sink, rather than
> > the
> > > > > > > source,
> > > > > > > > so
> > > > > > > > > > > >> that
> > > > > > > > > > > >>>>> it includes the processing latency of the task
> > itself,
> > > > > not
> > > > > > > just
> > > > > > > > > the
> > > > > > > > > > > >>>> latency
> > > > > > > > > > > >>>>> of everything up to, but not including, the task
> > (which
> > > > > > seems
> > > > > > > > > > > >> confusing
> > > > > > > > > > > >>>>> for users). I guess this could also be an
> > > > implementation
> > > > > > > > detail,
> > > > > > > > > > > >>> though.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > > > >>>>> -John
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman
> > > > > wrote:
> > > > > > > > > > > >>>>>> Hey all,
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> After discussing with Bruno I'd like to propose a
> > > > small
> > > > > > > > > amendment,
> > > > > > > > > > > >>>>>> which is to record the processor-node-level
> > metrics
> > > > only
> > > > > > for
> > > > > > > > > > > >>> *stateful*
> > > > > > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > > > > > "processor-node-level"
> > > > > > > > > > > >>>>>> metric and not a "state-store-level" metric as the
> > > > > > staleness
> > > > > > > > is
> > > > > > > > > > > >> still
> > > > > > > > > > > >>>>>> a property of the node rather than of the state
> > > > itself.
> > > > > > > > > However, it
> > > > > > > > > > > >>>> seems
> > > > > > > > > > > >>>>>> that this information is primarily useful for
> > stateful
> > > > > > > > operators
> > > > > > > > > > > >> that
> > > > > > > > > > > >>>> might
> > > > > > > > > > > >>>>>> be exposing state via IQ or otherwise dependent
> > on the
> > > > > > > record
> > > > > > > > > time
> > > > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> It's worth calling out that recent performance
> > > > > > improvements
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >>>> metrics
> > > > > > > > > > > >>>>>> framework mean that we no longer fetch the system
> > time
> > > > > at
> > > > > > > the
> > > > > > > > > > > >> operator
> > > > > > > > > > > >>>>>> level, but only once per task. In other words the
> > > > system
> > > > > > > time
> > > > > > > > > is not
> > > > > > > > > > > >>>> updated
> > > > > > > > > > > >>>>>> between each process as a record flows through the
> > > > > > > > subtopology,
> > > > > > > > > so
> > > > > > > > > > > >>>>>> debugging the processor-level latency via the
> > > > > stateleness
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > >> be
> > > > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > > > operator-level
> > > > > > > > metrics
> > > > > > > > > are
> > > > > > > > > > > >>> not
> > > > > > > > > > > >>>>>> *useful* relative to the task-level metric.
> > Upstream
> > > > > > caching
> > > > > > > > > and/or
> > > > > > > > > > > >>>>>> suppression
> > > > > > > > > > > >>>>>> can still cause a record's staleness at some
> > > > downstream
> > > > > > > > stateful
> > > > > > > > > > > >>>> operator
> > > > > > > > > > > >>>>>> to deviate from the task-level staleness
> > (recorded at
> > > > > the
> > > > > > > > source
> > > > > > > > > > > >>> node).
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> Please let me know if you have any concerns about
> > this
> > > > > > > change.
> > > > > > > > > The
> > > > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > > > > > > bruno@confluent.io>
> > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > > > >>>>>>> I think we do not need it now. If we need it, we
> > can
> > > > > add
> > > > > > > it.
> > > > > > > > > Here
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>> old truism applies: Adding is always easier than
> > > > > > removing.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > > > >>>>>>> I think it is good to have the staleness metrics
> > also
> > > > > on
> > > > > > > > > > > >>>>>>> processor-node-level. If we do not want to record
> > > > them
> > > > > on
> > > > > > > all
> > > > > > > > > > > >>>>>>> processor nodes, you could restrict the
> > recording to
> > > > > > > stateful
> > > > > > > > > > > >>>>>>> processor-nodes, since those are the ones that
> > would
> > > > > > > benefit
> > > > > > > > > most
> > > > > > > > > > > >>> from
> > > > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Best,
> > > > > > > > > > > >>>>>>> Bruno
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie
> > Blee-Goldman <
> > > > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to align
> > with the
> > > > > > > current
> > > > > > > > > > > >>> metrics.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Is it better to conform than to be right?
> > History
> > > > has
> > > > > a
> > > > > > > lot
> > > > > > > > > to say
> > > > > > > > > > > >>> on
> > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > >>>>>>>> matter
> > > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies to the
> > fine
> > > > > > > details
> > > > > > > > of
> > > > > > > > > > > >>> metrics
> > > > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> More seriously, I figured if people are looking
> > at
> > > > > this
> > > > > > > > metric
> > > > > > > > > > > >>> they're
> > > > > > > > > > > >>>>>>>> likely to
> > > > > > > > > > > >>>>>>>> be looking at all the others. Then naming this
> > one
> > > > > > "-mean"
> > > > > > > > > would
> > > > > > > > > > > >>>> probably
> > > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in
> > the
> > > > > > other
> > > > > > > > > metrics
> > > > > > > > > > > >>> has
> > > > > > > > > > > >>>> a
> > > > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and
> > > > p75)
> > > > > > > > better.
> > > > > > > > > I'll
> > > > > > > > > > > >>>> swap
> > > > > > > > > > > >>>>>>>> that out
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > > > > > > vvcephei@apache.org
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a
> > > > > reason
> > > > > > to
> > > > > > > > > choose
> > > > > > > > > > > >>>> "avg"
> > > > > > > > > > > >>>>>>>>> instead
> > > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I
> > might
> > > > be
> > > > > > > > > > > >> oversensitive
> > > > > > > > > > > >>>>>>> because
> > > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but it
> > strikes
> > > > me
> > > > > > > that
> > > > > > > > > "avg"
> > > > > > > > > > > >> is
> > > > > > > > > > > >>>>>>>>> actually
> > > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of
> > statistics,
> > > > > > > whereas
> > > > > > > > > "mean"
> > > > > > > > > > > >>> is
> > > > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none
> > with
> > > > > > > "mean";
> > > > > > > > > was
> > > > > > > > > > > >>> that
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Regarding the names of the percentile, I
> > actually
> > > > > > > couldn't
> > > > > > > > > find
> > > > > > > > > > > >>> _any_
> > > > > > > > > > > >>>>>>> other
> > > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there a
> > reason to
> > > > > > choose
> > > > > > > > > "99th"
> > > > > > > > > > > >> as
> > > > > > > > > > > >>>>>>> opposed
> > > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> > > > > criticism,
> > > > > > > I'm
> > > > > > > > > just
> > > > > > > > > > > >>>>>>> primarily
> > > > > > > > > > > >>>>>>>>> asking
> > > > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > > >>>>>>>>> -John
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie
> > Blee-Goldman
> > > > > > > wrote:
> > > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min
> > metric
> > > > for
> > > > > > the
> > > > > > > > > sake of
> > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or
> > 50th
> > > > > > > > > percentile)
> > > > > > > > > > > >>> but I
> > > > > > > > > > > >>>>>>> can
> > > > > > > > > > > >>>>>>>>> see
> > > > > > > > > > > >>>>>>>>>> how users might expect it and find it a bit
> > > > > > disorienting
> > > > > > > > > not to
> > > > > > > > > > > >>>>>>> have. So
> > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > > > percentile)*
> > > > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > > > percentile)*
> > > > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to
> > embarrass
> > > > > > > myself.
> > > > > > > > I
> > > > > > > > > > > >> guess
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>> mean
> > > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the
> > median, but
> > > > > > > neither
> > > > > > > > > are
> > > > > > > > > > > >> as
> > > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th
> > and
> > > > > max).
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but
> > it
> > > > > does
> > > > > > > > round
> > > > > > > > > out
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>>>> mental
> > > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always
> > graphed
> > > > > min
> > > > > > > > along
> > > > > > > > > with
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > >>>>>>>>>>> metrics to help me understand how fast the
> > system
> > > > > can
> > > > > > > be,
> > > > > > > > > which
> > > > > > > > > > > >>>>>>> helps
> > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a
> > relatively
> > > > > > > > inexpensive
> > > > > > > > > > > >> metric
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just throw
> > it in.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> > > > Blee-Goldman
> > > > > > > > wrote:
> > > > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end
> > latency
> > > > > *up*
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that
> > drives
> > > > > the
> > > > > > > > point
> > > > > > > > > > > >> home
> > > > > > > > > > > >>>>>>>>> better.
> > > > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is
> > received by
> > > > > that
> > > > > > > > > task" --
> > > > > > > > > > > >>>>>>> will
> > > > > > > > > > > >>>>>>>>> update
> > > > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the
> > min
> > > > > would
> > > > > > > > > actually
> > > > > > > > > > > >> be
> > > > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't
> > feel
> > > > > > strongly
> > > > > > > > > that we
> > > > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can
> > you
> > > > > > > > enlighten
> > > > > > > > > me
> > > > > > > > > > > >>>>>>> with an
> > > > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the
> > overhead
> > > > of
> > > > > > > > adding
> > > > > > > > > > > >>>>>>> multiple
> > > > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate
> > what
> > > > kind
> > > > > > of
> > > > > > > > > > > >>> performance
> > > > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th
> > > > > percentile
> > > > > > > > > would be
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>>>> median
> > > > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean
> > *and*
> > > > the
> > > > > > 50th
> > > > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John
> > Roesler <
> > > > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so
> > far:
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether
> > we
> > > > > report
> > > > > > > > > "spans"
> > > > > > > > > > > >> or
> > > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice,
> > some of
> > > > > the
> > > > > > > > spans
> > > > > > > > > are
> > > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like time
> > spent
> > > > > > waiting
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > > > >>>>>>>>>>>>> time from the event happening to the ETL
> > > > producer
> > > > > > > > > choosing to
> > > > > > > > > > > >>>>>>> send
> > > > > > > > > > > >>>>>>>>> it,
> > > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers,
> > etc.,
> > > > etc.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically easier to
> > > > > compute
> > > > > > > > spans
> > > > > > > > > by
> > > > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e
> > > > latencies
> > > > > > by
> > > > > > > > > adding
> > > > > > > > > > > >>>>>>> spans.
> > > > > > > > > > > >>>>>>>>> You
> > > > > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> > > > computation
> > > > > > from
> > > > > > > > e2e
> > > > > > > > > > > >>>>>>> always
> > > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas
> > > > > computing
> > > > > > > e2e
> > > > > > > > > > > >> latency
> > > > > > > > > > > >>>>>>>>> from
> > > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans
> > leading
> > > > up
> > > > > to
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > > >> you
> > > > > > > > > > > >>>>>>>>> care
> > > > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to have
> > > > spans
> > > > > > when
> > > > > > > > > they
> > > > > > > > > > > >> are
> > > > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a
> > more
> > > > > > > general
> > > > > > > > > > > >>>>>>> measurement
> > > > > > > > > > > >>>>>>>>>>>>> that basically every person/application
> > cares
> > > > > about
> > > > > > > and
> > > > > > > > > > > >> should
> > > > > > > > > > > >>>>>>> be
> > > > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more
> > > > value
> > > > > to
> > > > > > > > more
> > > > > > > > > > > >>>>>>> people if
> > > > > > > > > > > >>>>>>>>> we
> > > > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding
> > "record-staleness" as
> > > > a
> > > > > > > name,
> > > > > > > > I
> > > > > > > > > > > >> think
> > > > > > > > > > > >>>>>>> I
> > > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> > > > > intuition.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice,
> > > > since
> > > > > > the
> > > > > > > > > inside
> > > > > > > > > > > >> of
> > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > >>>>>>>>> task
> > > > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of latency
> > in
> > > > some
> > > > > > > > cases.
> > > > > > > > > > > >> Plus,
> > > > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to
> > > > know
> > > > > > (for
> > > > > > > > the
> > > > > > > > > > > >>>>>>> purposes
> > > > > > > > > > > >>>>>>>>> of
> > > > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes source
> > events
> > > > to
> > > > > > > "show
> > > > > > > > > up" at
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at every
> > > > processor
> > > > > > > could
> > > > > > > > be
> > > > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the
> > node
> > > > > > level.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record
> > also.
> > > > > > Even
> > > > > > > if
> > > > > > > > > it's
> > > > > > > > > > > >> a
> > > > > > > > > > > >>>>>>> poor
> > > > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people
> > might
> > > > > > > expect
> > > > > > > > > to see
> > > > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there
> > a
> > > > > reason
> > > > > > > to
> > > > > > > > > leave
> > > > > > > > > > > >> it
> > > > > > > > > > > >>>>>>>>> off?
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > > > >>>>>>>>>>>>> I agree we should include late records
> > (though
> > > > > not
> > > > > > > the
> > > > > > > > > ones
> > > > > > > > > > > >> we
> > > > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> > > > > > legitimately
> > > > > > > > > some
> > > > > > > > > > > >>>>>>> records
> > > > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole
> > > > point
> > > > > > of
> > > > > > > > > these
> > > > > > > > > > > >>>>>>> metrics.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other
> > > > > feedback,
> > > > > > > > other
> > > > > > > > > > > >> than
> > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang
> > Wang
> > > > > > wrote:
> > > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick
> > > > thoughts:
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both
> > the
> > > > > > > processing
> > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in
> > intermediate
> > > > > > > topics.
> > > > > > > > I
> > > > > > > > > have
> > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the
> > > > > latency
> > > > > > > > > metric of
> > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > >>>>>>>>> task A
> > > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > > > > sub-topology
> > > > > > > > up-to
> > > > > > > > > but
> > > > > > > > > > > >>>>>>> not
> > > > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name
> > > > > "latency"
> > > > > > > > > itself,
> > > > > > > > > > > >>>>>>> since
> > > > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics
> > already
> > > > > > which
> > > > > > > > are
> > > > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc,
> > > > while
> > > > > > > here
> > > > > > > > > we are
> > > > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the record
> > > > timestamp.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > > > > "record-staleness"
> > > > > > > > > > > >>>>>>> (note we
> > > > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which
> > > > case
> > > > > > > > > recording at
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>> system-time before we start processing the
> > > > > record
> > > > > > > > sounds
> > > > > > > > > > > >> more
> > > > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we
> > already
> > > > > > have a
> > > > > > > > > > > >>>>>>> task-level
> > > > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific
> > node's
> > > > e2e
> > > > > > > > > latency is
> > > > > > > > > > > >>>>>>>>> similar
> > > > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level
> > processing
> > > > > > > > latency.
> > > > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness
> > metric is
> > > > > > > > > sufficient.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > > > > > > Blee-Goldman <
> > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly
> > > > > useful
> > > > > > > > gauge
> > > > > > > > > for
> > > > > > > > > > > >>>>>>> this
> > > > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an
> > > > > *upper
> > > > > > > > > *bound on
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose
> > percentiles
> > > > > > that
> > > > > > > > > would
> > > > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will
> > experience,
> > > > > and
> > > > > > > > what
> > > > > > > > > > > >>>>>>> *close
> > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific
> > > > > numbers
> > > > > > > and
> > > > > > > > > > > >>>>>>> could be
> > > > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing
> > > > from
> > > > > > > users
> > > > > > > > on
> > > > > > > > > > > >>>>>>> this.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> > > > "hop-to-hop
> > > > > > > > > latency" in
> > > > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> > > > subtracting
> > > > > > the
> > > > > > > > > > > >>>>>>> previous
> > > > > > > > > > > >>>>>>>>> node's
> > > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do
> > it
> > > > > either
> > > > > > > way
> > > > > > > > > since
> > > > > > > > > > > >>>>>>>>> you can
> > > > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think
> > the
> > > > > > > > end-to-end
> > > > > > > > > > > >>>>>>> latency
> > > > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not
> > to
> > > > > debug
> > > > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > > > >>>>>>>>> in the
> > > > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it
> > takes
> > > > > > arecord
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this might
> > be
> > > > > useful
> > > > > > > for
> > > > > > > > > users
> > > > > > > > > > > >>>>>>>>> who are
> > > > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just
> > produced
> > > > > will
> > > > > > > be
> > > > > > > > > > > >>>>>>> included
> > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect --
> > but
> > > > > > maybe I
> > > > > > > > > didn't
> > > > > > > > > > > >>>>>>> make
> > > > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this
> > in
> > > > the
> > > > > > KIP.
> > > > > > > > The
> > > > > > > > > > > >>>>>>> short
> > > > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I added a
> > > > > paragraph
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >>>>>>> end
> > > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning
> > > > here,
> > > > > > > please
> > > > > > > > > let
> > > > > > > > > > > >>>>>>> me
> > > > > > > > > > > >>>>>>>>> know
> > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the
> > existing
> > > > > > metric
> > > > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > > > Node#process
> > > > > > > method
> > > > > > > > > to
> > > > > > > > > > > >>>>>>> run
> > > > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative to the
> > > > event
> > > > > > > > > timestamp.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is
> > > > pretty
> > > > > > > > > confusing
> > > > > > > > > > > >>>>>>> so
> > > > > > > > > > > >>>>>>>>> I've
> > > > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to
> > "end-to-end-latency"
> > > > > > which I
> > > > > > > > > feel
> > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang
> > Chen
> > > > <
> > > > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the
> > E2E
> > > > > > latency
> > > > > > > > is
> > > > > > > > > > > >>>>>>>>> important
> > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> > > > > application.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of
> > picking
> > > > > 75,
> > > > > > > 99
> > > > > > > > > and
> > > > > > > > > > > >>>>>>> max
> > > > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > > > > > > percentile
> > > > > > > > as
> > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > > > > > > cumulative,
> > > > > > > > > I.E
> > > > > > > > > > > >>>>>>> if a
> > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) =
> > T(B->C) +
> > > > > P(B)
> > > > > > =
> > > > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the
> > captured
> > > > > > > > latency,
> > > > > > > > > > > >>>>>>> and T()
> > > > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records
> > between
> > > > > two
> > > > > > > > nodes,
> > > > > > > > > > > >>>>>>>>> including
> > > > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe
> > having
> > > > > > T(B->C)
> > > > > > > > and
> > > > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> > > > > > otherwise
> > > > > > > > if
> > > > > > > > > > > >>>>>>> there
> > > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are
> > affected in
> > > > > the
> > > > > > > same
> > > > > > > > > > > >>>>>>> time.
> > > > > > > > > > > >>>>>>>>> In the
> > > > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful
> > only
> > > > > when
> > > > > > > the
> > > > > > > > > > > >>>>>>> record
> > > > > > > > > > > >>>>>>>>> exits
> > > > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this
> > > > record
> > > > > > > spent
> > > > > > > > > > > >>>>>>> inside
> > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate
> > treatment
> > > > for
> > > > > > > sink
> > > > > > > > > > > >>>>>>> nodes and
> > > > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count
> > the
> > > > time
> > > > > > > > > receiving
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution
> > here,
> > > > > > just
> > > > > > > > > want to
> > > > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late
> > arrival
> > > > > > records
> > > > > > > > as
> > > > > > > > > > > >>>>>>> well,
> > > > > > > > > > > >>>>>>>>> they
> > > > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > > > > > > out-of-order
> > > > > > > > > > > >>>>>>>>> records are
> > > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records.
> > Should we
> > > > > > also
> > > > > > > > > supply
> > > > > > > > > > > >>>>>>> a
> > > > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should
> > just
> > > > > take
> > > > > > > > care
> > > > > > > > > of
> > > > > > > > > > > >>>>>>> it
> > > > > > > > > > > >>>>>>>>> by
> > > > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we
> > haven't
> > > > > > > discussed
> > > > > > > > > its
> > > > > > > > > > > >>>>>>>>> relation
> > > > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics,
> > could
> > > > you
> > > > > > add
> > > > > > > > > some
> > > > > > > > > > > >>>>>>>>> context
> > > > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use
> > which`
> > > > > > > tutorial
> > > > > > > > > for
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > > > > > > Blee-Goldman
> > > > > > > > <
> > > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on
> > KIP-613
> > > > > > which
> > > > > > > > aims
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>>> add
> > > > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please
> > take a
> > > > > look:
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
> --
> -- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
We do have a big JIRA for making sensor recording to be sufficient. But it
is a bit hard to try to reduce the num.system clock calls in general (but
admittedly, in nowadays OS they are supposed to be very cheap as well),
I've just recently merged a small PR trying to make less
SystemTime.currentMilliseconds calls whenever possible, but it is hard to
enforce that for all callers.

Guozhang

On Sat, May 16, 2020 at 9:46 AM John Roesler <vv...@apache.org> wrote:

> FWIW, I’d be fine with TRACE, if it makes the processor-level metrics more
> palatable to everyone.
>
> Ultimately, I think this is just kicking the can down the road, and we
> should really just refactor our sensors/metrics to be lower overhead. But
> that’s too much work to consider right before the release so for now, we
> can kick that can just a little farther.
>
> Thanks for proposing a solution, Guozhang!
>
> -John
>
> On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> > Cool, that's good points -- we should not let our current metrics
> proposal
> > to depend on future roadmaps, since they can still be removed later if
> ever
> > becoming not valuable after all.
> >
> > On the other hand, what do you think about introducing TRACE level
> metrics
> > recording and make the the processor-node level metrics on TRACE?
> >
> > Guozhang
> >
> > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <sophie@confluent.io
> >
> > wrote:
> >
> > > Yeah, I had proposed to replace the "source + sink" task-level metric
> > > with "sink only" task-level metrics plus the stateful node-level ones.
> > >
> > > I completely agree that this proposal should and would look different
> > > once we complete those two proposals, ie decouple caching and
> > > consolidate suppression. But as it is right now, caching and
> suppression
> > > cause the actual time-of-processing of records to vary greatly
> throughout
> > > a subtopology such that the delta between a source node and an
> intermediate
> > > one is potentially quite large.
> > >
> > > For this exact reason I would argue that this metric is still valuable
> at
> > > the node-level
> > >  even if the "current time" is actually only updated once for a task.
> > > Obviously this
> > > would not convey any information about the actual processing latency,
> but
> > > as you
> > > stated, this is likely to be a small delta on the scale of
> milliseconds.
> > > But the latency
> > > due to a suppression can be on the order of minutes, hours, or even
> days!
> > > If
> > > there is more than one suppression or caching in a subtopology, the
> source
> > > and sink
> > > e2e latency could be wildly misrepresentative of the e2e latency of a
> node
> > > in the middle.
> > >
> > > Anyways, I removed this last point from the current KIP document
> because I
> > > felt
> > > it was ultimately an implementation detail. But the larger point I
> would
> > > argue is that:
> > > if we are truly worried about the performance hit of fetching the
> system
> > > time at every
> > > node, we should just not fetch the system time at every node instead of
> > > dropping the
> > > metric altogether. It may be slightly off but only on the order of ms
> and
> > > not on the order
> > > of hours, and it's still more useful than no metric at all
> > >
> > >
> > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> > >
> > > > Hi Sophie,
> > > >
> > > > I think your motivation makes sense, in fact before KIP-444 we used
> to
> > > have
> > > > processor-node-level process latency as DEBUG which would
> holistically
> > > > address the motivation but we removed it since it is indeed quite
> > > expensive
> > > > (I do not have concrete numbers at hand, but basically you'd have to
> call
> > > > system-wall-clock time a couple of time for each record, each
> processor
> > > > node), and in the long run I feel such scenarios would be less
> common if
> > > we
> > > > 1) decouple caching from emitting and 2) always put suppression right
> > > > before sink only, and that's also why I've proposed to remove them in
> > > > KIP-444.
> > > >
> > > > I missed an update in your wiki that right now you've updated the
> > > > task-level metric to be recorded at the sink node, and thought it is
> > > still
> > > > at the source node, but assuming it is still the case, then
> practically
> > > any
> > > > state store's staleness of the task could roughly be measured as that
> > > > task-level staleness plus a small delta (usually tens of mills), and
> the
> > > > down stream task B's staleness - up stream task A's staleness would
> > > capture
> > > > the suppression effect at the end of A plus the intermediate topic
> > > > producing -> consuming latency, which usually would not require to be
> > > > distinguished.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <
> sophie@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Hey Guozhang,
> > > > >
> > > > > Thanks for the response. I meant to ask this earlier and forgot,
> but do
> > > > you
> > > > > have any data or benchmarking results on hand for the performance
> hit
> > > we
> > > > > took due to the metrics? Or maybe better yet, on the performance we
> > > > gained
> > > > > back by dropping some metrics? I think it would help us to better
> > > > > understand
> > > > > the tradeoffs with some concrete numbers to compare.
> > > > >
> > > > > I guess I'm inclined to want to include these metrics at the
> operator
> > > > level
> > > > > on
> > > > > DEBUG since, in my mind, that's exactly what DEBUG is for:
> performance-
> > > > > focused users will presumably run at INFO level and even those who
> do
> > > > > use DEBUG may only do so temporarily, when they actually have
> something
> > > > > to debug.
> > > > >
> > > > > Of course, I'm not running a Streams app myself so all this is just
> > > > > conjecture.
> > > > > But it seems reasonable to interpret the two metrics levels as
> "minimal
> > > > > useful
> > > > > information" and "all information that provides insight into the
> > > system"
> > > > >
> > > > > If we can agree there, the question becomes "does this provide any
> > > > > additional
> > > > > insight?" I would still argue that it does. To give a concrete
> example,
> > > > > let's say
> > > > > the user has a basic
> > > > >
> > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > > >
> > > > > subtopology, the application performing IQ on the transformer's
> state.
> > > > The
> > > > > user
> > > > > wants to know how long it's taking records to be reflected in the
> > > results
> > > > > of a query.
> > > > > With only task-level metrics, they can only guess based on the e2e
> > > > latency
> > > > > of the
> > > > > previous task and of this one. But neither of those is a good
> > > > > representation of the
> > > > > time it will take a record to be processed by the transformer: the
> > > > previous
> > > > > task's
> > > > > latency will not include the time for the record to go through the
> > > > > repartitioning,
> > > > > and the current task's latency is measured after a suppression. The
> > > > > suppression
> > > > > itself will introduce significant latency and doesn't provide an
> > > accurate
> > > > > estimate.
> > > > > Caching would also do what it does best, messing with our
> understanding
> > > > of
> > > > > time :)
> > > > >
> > > > > It also means you can't measure the consumption latency, so there
> would
> > > > be
> > > > > no way to tell if the e2e latency is so high because the very first
> > > > > subtopology is
> > > > > extremely slow, or the record was severely delayed in reaching the
> > > input
> > > > > topic.
> > > > > I guess my earlier proposal to include both source and sink metrics
> > > could
> > > > > address this issue, but it seems to solve the problem more
> holistically
> > > > --
> > > > > and for
> > > > > more complex subtopologies -- by including metrics at the processor
> > > level
> > > > >
> > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hey folks,
> > > > > >
> > > > > > I want to make a final fight on processor-node-level metrics :)
> Post
> > > > > > KIP-444 we've actually removed a lot of node-level metrics since
> it
> > > is
> > > > > too
> > > > > > expensive to record at that level and their values are not proven
> > > worth
> > > > > it.
> > > > > > Again I'd use the "we have to either enable all or none of DEBUG
> > > > metrics"
> > > > > > card, e.g. if we want to look into per-task process-latency
> which is
> > > > > DEBUG,
> > > > > > we'd have to record all process-node / state-store / cache level
> > > > metrics.
> > > > > > On the other hand, within a task we usually do not have a lot of
> > > > stateful
> > > > > > operator nodes and if we want to find if certain stateful nodes
> are
> > > the
> > > > > > bottleneck of process latency, that is usually discoverable from
> the
> > > > > > state-store level read / write latencies already. So I'd imagine
> that
> > > > > > suppose there's a sub-topology with multiple state stores in it,
> if
> > > we
> > > > > > found the task-level process latency is high, we can still tell
> which
> > > > > state
> > > > > > stores within it is the main bottleneck from the state store
> read /
> > > > write
> > > > > > latency (I'm assuming the IO latency is always dominant compared
> with
> > > > > > others).
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > > sophie@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > > @Matthias
> > > > > > > > And that's what he says about the 95th percentile! Imagine
> what
> > > he
> > > > > > would
> > > > > > > say about the 50th :P
> > > > > > >
> > > > > > > I should have kept watching another 10 minutes. He gets around
> to
> > > > > > covering
> > > > > > > the 50th, and let's
> > > > > > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <
> vvcephei@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks, Sophie!
> > > > > > > >
> > > > > > > > I think this makes perfect sense. It will be much more
> intuitive
> > > to
> > > > > use
> > > > > > > > the metric for the stated motivation this way.  I’d be in
> favor
> > > of
> > > > > the
> > > > > > > > proposal after this update.
> > > > > > > >
> > > > > > > > Thanks again for taking this on,
> > > > > > > > -John
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to clarify/modify one aspect of this KIP, which
> is to
> > > > > record
> > > > > > > the
> > > > > > > > > staleness
> > > > > > > > > at the *completion* of the record's processing by the
> > > > operator/task
> > > > > > in
> > > > > > > > > question,
> > > > > > > > > rather than on the intake. The task-level metrics will be
> > > > recorded
> > > > > at
> > > > > > > the
> > > > > > > > > sink
> > > > > > > > > node instead of at the source, and the operator-level
> metrics
> > > > will
> > > > > be
> > > > > > > > > recorded
> > > > > > > > > at the end of the operation.
> > > > > > > > >
> > > > > > > > > The stated purpose and intended usefulness of this KIP is
> to
> > > give
> > > > > > > users a
> > > > > > > > > way
> > > > > > > > > to gauge roughly how long it takes for a record to be
> reflected
> > > > in
> > > > > > the
> > > > > > > > > "results",
> > > > > > > > > whether these results are being read from an output topic
> or
> > > > > through
> > > > > > > IQ.
> > > > > > > > To
> > > > > > > > > take the IQ example, the results of a record are obviously
> not
> > > > > > visible
> > > > > > > > until
> > > > > > > > > *after* that node has finished processing it. The
> staleness,
> > > > while
> > > > > > > still
> > > > > > > > > potentially
> > > > > > > > > useful as it can impact the *way* a record is processed in
> a
> > > > > stateful
> > > > > > > and
> > > > > > > > > time-
> > > > > > > > > dependent operator, is not part of the problem this KIP
> > > > > specifically
> > > > > > > set
> > > > > > > > out
> > > > > > > > > to solve.
> > > > > > > > >
> > > > > > > > > In light of this I think it's appropriate to revert the
> name
> > > > change
> > > > > > > back
> > > > > > > > to
> > > > > > > > > include
> > > > > > > > > latency, since "staleness" as described above only makes
> sense
> > > > when
> > > > > > > > > measuring
> > > > > > > > > relative to the arrival of a record at a task/node. I'd
> propose
> > > > to
> > > > > > save
> > > > > > > > the
> > > > > > > > > term
> > > > > > > > > "staleness" for that particular meaning, and adopt
> Matthias's
> > > > > > > suggestion
> > > > > > > > of
> > > > > > > > > "record-e2e-latency" for this.
> > > > > > > > >
> > > > > > > > > Thanks for hanging in there all. Please let me know if you
> have
> > > > any
> > > > > > > > > concerns about
> > > > > > > > > this change!
> > > > > > > > >
> > > > > > > > > Sophie
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > > mjsax@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I am also happy with max/min/99/90. And I buy your naming
> > > > > argument
> > > > > > > > about
> > > > > > > > > > staleness vs latency.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -Matthias
> > > > > > > > > >
> > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > > Hey Sophie,
> > > > > > > > > > >
> > > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > > > > > sophie@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> @Matthias
> > > > > > > > > > >> Regarding tracking the 50th percentile, I'll refer
> you to
> > > > the
> > > > > > 4:53
> > > > > > > > mark
> > > > > > > > > > of
> > > > > > > > > > >> the video
> > > > > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > > >>
> > > > > > > > > > >> And that's what he says about the 95th percentile!
> Imagine
> > > > > what
> > > > > > he
> > > > > > > > would
> > > > > > > > > > >> say about
> > > > > > > > > > >> the 50th :P
> > > > > > > > > > >>
> > > > > > > > > > >> But seriously, since we can't seem to agree that the
> mean
> > > or
> > > > > > 50th
> > > > > > > > > > >> percentile is actually
> > > > > > > > > > >> useful I'm inclined to resurrect my original proposal,
> > > > > neither.
> > > > > > > But
> > > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > >> that's a good
> > > > > > > > > > >> argument against the 75th, which I admittedly chose
> > > somewhat
> > > > > > > > > > arbitrarily as
> > > > > > > > > > >> an
> > > > > > > > > > >> intermediate between the 50th and the higher
> percentiles.
> > > > How
> > > > > > > about:
> > > > > > > > > > >>
> > > > > > > > > > >> -max
> > > > > > > > > > >> -p99
> > > > > > > > > > >> -p90
> > > > > > > > > > >> -min
> > > > > > > > > > >>
> > > > > > > > > > >> with p50/mean still up for debate if anyone feels
> strongly
> > > > for
> > > > > > > > either of
> > > > > > > > > > >> them.
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding the name, I've already flip-flopped on this
> so
> > > I'm
> > > > > > > > definitely
> > > > > > > > > > >> still open to
> > > > > > > > > > >> further arguments. But the reason for changing it from
> > > > > > > > > > end-to-end-latency
> > > > > > > > > > >> (which
> > > > > > > > > > >> is similar to what you propose) is that this metric
> > > > > technically
> > > > > > > > reflects
> > > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > > >> the record is when it's *received* by the operator,
> not
> > > when
> > > > > > it's
> > > > > > > > > > processed
> > > > > > > > > > >> by the operator.
> > > > > > > > > > >> It seemed like there was the potential for confusion
> that
> > > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > > >> represent the latency from the event creation to the
> time
> > > > the
> > > > > > > > processor
> > > > > > > > > > is
> > > > > > > > > > >> done
> > > > > > > > > > >> processing it.
> > > > > > > > > > >>
> > > > > > > > > > >> @John
> > > > > > > > > > >> I'd rather err on the side of "not-enough" metrics as
> we
> > > can
> > > > > > > always
> > > > > > > > add
> > > > > > > > > > >> this to the
> > > > > > > > > > >> stateless metrics later on. If we decide to measure
> the
> > > time
> > > > > at
> > > > > > > > every
> > > > > > > > > > node
> > > > > > > > > > >> and don't
> > > > > > > > > > >> find any evidence of a serious performance impact, and
> > > users
> > > > > > > > indicate
> > > > > > > > > > they
> > > > > > > > > > >> would
> > > > > > > > > > >> like to see this metric at all nodes, then we can
> easily
> > > > start
> > > > > > > > reporting
> > > > > > > > > > >> them as well.
> > > > > > > > > > >> WDYT?
> > > > > > > > > > >>
> > > > > > > > > > >> That said, sink nodes seem like a reasonable
> exception to
> > > > the
> > > > > > > rule.
> > > > > > > > > > >> Obviously users
> > > > > > > > > > >> should be able to detect the time when the record
> reaches
> > > > the
> > > > > > > output
> > > > > > > > > > topic
> > > > > > > > > > >> but that
> > > > > > > > > > >> still leaves a gap in understanding how long the
> > > production
> > > > > > > latency
> > > > > > > > was.
> > > > > > > > > > >> This mirrors
> > > > > > > > > > >> the consumption latency that is exposed by the
> task-level
> > > > > > metrics,
> > > > > > > > which
> > > > > > > > > > >> are measured
> > > > > > > > > > >> at the source node. For good symmetry what if we
> actually
> > > > > expose
> > > > > > > > both
> > > > > > > > > > the
> > > > > > > > > > >> source
> > > > > > > > > > >> and sink latency at the task-level? ie report both
> sets of
> > > > > > > > statistical
> > > > > > > > > > >> measurements with
> > > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > > >>
> > > > > > > > > > >> @Bill
> > > > > > > > > > >> Thanks for the comment about regarding the min! I
> hadn't
> > > > > > > considered
> > > > > > > > that
> > > > > > > > > > >> and it's
> > > > > > > > > > >> quite useful to think about how and what is useful
> from a
> > > > > users
> > > > > > > > point of
> > > > > > > > > > >> view.
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding your second. point, I'm inclined to leave
> that
> > > as
> > > > an
> > > > > > > > > > >> implementation detail
> > > > > > > > > > >> but my take would be that the user should be allowed
> to
> > > > > control
> > > > > > > the
> > > > > > > > > > record
> > > > > > > > > > >> timestamp
> > > > > > > > > > >> used for this with the timestamp extractor. My
> impression
> > > is
> > > > > > that
> > > > > > > > users
> > > > > > > > > > may
> > > > > > > > > > >> often embed
> > > > > > > > > > >> the actual event time in the payload for whatever
> reason,
> > > > and
> > > > > > this
> > > > > > > > > > >> represents the "true"
> > > > > > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > > bbejeck@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful
> metric
> > > to
> > > > > > add.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Regarding tracking min, I  think it could be
> valuable for
> > > > > users
> > > > > > > to
> > > > > > > > > > >> discern
> > > > > > > > > > >>> which part of their topologies are more efficient
> since
> > > > this
> > > > > > is a
> > > > > > > > > > >>> task-level metric.  I realize everyone seems to be on
> > > board
> > > > > > with
> > > > > > > > > > >> including
> > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on this
> topic
> > > > > should
> > > > > > > we
> > > > > > > > > > decide
> > > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > > >>>
> > > > > > > > > > >>> I do have a question regarding the calculation of
> > > > staleness.
> > > > > > > > > > >>> Is there going to be a consideration for timestamp
> > > > > extractors?
> > > > > > > > Users
> > > > > > > > > > >> could
> > > > > > > > > > >>> prefer to use a timestamp embedded in the payload,
> and it
> > > > > could
> > > > > > > > skew
> > > > > > > > > > the
> > > > > > > > > > >>> measurements.
> > > > > > > > > > >>> I was wondering if we should specify in the KIP if
> > > setting
> > > > > the
> > > > > > > > arrival
> > > > > > > > > > >> time
> > > > > > > > > > >>> is always going to come from the record timestamp,
> or is
> > > > this
> > > > > > an
> > > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks!
> > > > > > > > > > >>> Bill
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > > > > mjsax@apache.org>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think it's not useful to record the avg/mean; it
> > > > sensitive
> > > > > > to
> > > > > > > > > > >>>> outliers. We should rather track the median (50th
> > > > > percentile).
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Not sure if tracking min is useful, but I am also
> ok to
> > > > > track
> > > > > > > it.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> However, I find it odd to track 75th percentile.
> > > Standard
> > > > > > > measures
> > > > > > > > > > >> would
> > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need both, so
> maybe
> > > > > > picking
> > > > > > > > 90th
> > > > > > > > > > >>>> might be more useful?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> About the name: "staleness" wound really odd, and if
> > > fact
> > > > > the
> > > > > > > > metric
> > > > > > > > > > >>>> does capture "latency" so we should call it
> "latency". I
> > > > > > > > understand
> > > > > > > > > > the
> > > > > > > > > > >>>> issue that we already have a latency metric. So
> maybe we
> > > > > could
> > > > > > > > call it
> > > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> While I agree that we should include out-or-order
> data
> > > > (the
> > > > > > KIP
> > > > > > > > should
> > > > > > > > > > >>>> talk about `out-of-order` data, not `late` data;
> data is
> > > > > only
> > > > > > > > `late`
> > > > > > > > > > if
> > > > > > > > > > >>>> it's out-of-order and if it's dropped), I don't
> really
> > > > > > > understand
> > > > > > > > why
> > > > > > > > > > >>>> the new metric would help to configure grace period
> or
> > > > > > retention
> > > > > > > > time?
> > > > > > > > > > >>>> As you mention in the KIP, both are define as max
> > > > difference
> > > > > > of
> > > > > > > > > > >>>> `event-time - stream-time` and thus the new metric
> that
> > > > > takes
> > > > > > > > > > >>>> system-/wallclock-time into account does not seem to
> > > help
> > > > at
> > > > > > > all.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> > > > > Latency"
> > > > > > by
> > > > > > > > Gil
> > > > > > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> -Matthias
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> It seems like there would still be plenty of use
> cases
> > > > for
> > > > > > > > recording
> > > > > > > > > > >>>>> this metric at all processors and not just stateful
> > > ones,
> > > > > but
> > > > > > > I'm
> > > > > > > > > > >> happy
> > > > > > > > > > >>>>> to suspend my arguments for now. Since you're
> proposing
> > > > to
> > > > > > keep
> > > > > > > > > > >>>>> them at the processor-node level, it will be
> seamless
> > > > later
> > > > > > to
> > > > > > > > add
> > > > > > > > > > >>>>> in the stateless processors if we want. As a wise
> man
> > > > once
> > > > > > > said,
> > > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Regarding the time measurement, it's an
> implementation
> > > > > detail
> > > > > > > > > > >>>>> we don't need to consider in the KIP.
> Nevertheless, I'd
> > > > > > greatly
> > > > > > > > > > >>>>> prefer to measure the system time again when
> recording
> > > > the
> > > > > > > > > > >>>>> metric. I don't think we've seen any evidence that
> > > proves
> > > > > > this
> > > > > > > > > > >>>>> would harm performance, and the amount of
> inaccuracy
> > > > using
> > > > > > > > > > >>>>> the cached system time could incur is actually
> > > > substantial.
> > > > > > > But,
> > > > > > > > > > >>>>> if you want to just "not mention this" in the KIP,
> we
> > > can
> > > > > > defer
> > > > > > > > to
> > > > > > > > > > >>>>> the actual PR discussion, at which time we're in a
> > > better
> > > > > > > > position
> > > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Along the lines of the measurement accuracy
> discussion,
> > > > one
> > > > > > > > > > >>>>> minor thought I had is that maybe we should
> consider
> > > > > > measuring
> > > > > > > > > > >>>>> the task staleness metric at the sink, rather than
> the
> > > > > > source,
> > > > > > > so
> > > > > > > > > > >> that
> > > > > > > > > > >>>>> it includes the processing latency of the task
> itself,
> > > > not
> > > > > > just
> > > > > > > > the
> > > > > > > > > > >>>> latency
> > > > > > > > > > >>>>> of everything up to, but not including, the task
> (which
> > > > > seems
> > > > > > > > > > >> confusing
> > > > > > > > > > >>>>> for users). I guess this could also be an
> > > implementation
> > > > > > > detail,
> > > > > > > > > > >>> though.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > > >>>>> -John
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman
> > > > wrote:
> > > > > > > > > > >>>>>> Hey all,
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> After discussing with Bruno I'd like to propose a
> > > small
> > > > > > > > amendment,
> > > > > > > > > > >>>>>> which is to record the processor-node-level
> metrics
> > > only
> > > > > for
> > > > > > > > > > >>> *stateful*
> > > > > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > > > > "processor-node-level"
> > > > > > > > > > >>>>>> metric and not a "state-store-level" metric as the
> > > > > staleness
> > > > > > > is
> > > > > > > > > > >> still
> > > > > > > > > > >>>>>> a property of the node rather than of the state
> > > itself.
> > > > > > > > However, it
> > > > > > > > > > >>>> seems
> > > > > > > > > > >>>>>> that this information is primarily useful for
> stateful
> > > > > > > operators
> > > > > > > > > > >> that
> > > > > > > > > > >>>> might
> > > > > > > > > > >>>>>> be exposing state via IQ or otherwise dependent
> on the
> > > > > > record
> > > > > > > > time
> > > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> It's worth calling out that recent performance
> > > > > improvements
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >>>> metrics
> > > > > > > > > > >>>>>> framework mean that we no longer fetch the system
> time
> > > > at
> > > > > > the
> > > > > > > > > > >> operator
> > > > > > > > > > >>>>>> level, but only once per task. In other words the
> > > system
> > > > > > time
> > > > > > > > is not
> > > > > > > > > > >>>> updated
> > > > > > > > > > >>>>>> between each process as a record flows through the
> > > > > > > subtopology,
> > > > > > > > so
> > > > > > > > > > >>>>>> debugging the processor-level latency via the
> > > > stateleness
> > > > > > will
> > > > > > > > not
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > > operator-level
> > > > > > > metrics
> > > > > > > > are
> > > > > > > > > > >>> not
> > > > > > > > > > >>>>>> *useful* relative to the task-level metric.
> Upstream
> > > > > caching
> > > > > > > > and/or
> > > > > > > > > > >>>>>> suppression
> > > > > > > > > > >>>>>> can still cause a record's staleness at some
> > > downstream
> > > > > > > stateful
> > > > > > > > > > >>>> operator
> > > > > > > > > > >>>>>> to deviate from the task-level staleness
> (recorded at
> > > > the
> > > > > > > source
> > > > > > > > > > >>> node).
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Please let me know if you have any concerns about
> this
> > > > > > change.
> > > > > > > > The
> > > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > > > > > bruno@confluent.io>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > > >>>>>>> I think we do not need it now. If we need it, we
> can
> > > > add
> > > > > > it.
> > > > > > > > Here
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>>> old truism applies: Adding is always easier than
> > > > > removing.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > > >>>>>>> I think it is good to have the staleness metrics
> also
> > > > on
> > > > > > > > > > >>>>>>> processor-node-level. If we do not want to record
> > > them
> > > > on
> > > > > > all
> > > > > > > > > > >>>>>>> processor nodes, you could restrict the
> recording to
> > > > > > stateful
> > > > > > > > > > >>>>>>> processor-nodes, since those are the ones that
> would
> > > > > > benefit
> > > > > > > > most
> > > > > > > > > > >>> from
> > > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Best,
> > > > > > > > > > >>>>>>> Bruno
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie
> Blee-Goldman <
> > > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to align
> with the
> > > > > > current
> > > > > > > > > > >>> metrics.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Is it better to conform than to be right?
> History
> > > has
> > > > a
> > > > > > lot
> > > > > > > > to say
> > > > > > > > > > >>> on
> > > > > > > > > > >>>>>>> that
> > > > > > > > > > >>>>>>>> matter
> > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies to the
> fine
> > > > > > details
> > > > > > > of
> > > > > > > > > > >>> metrics
> > > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> More seriously, I figured if people are looking
> at
> > > > this
> > > > > > > metric
> > > > > > > > > > >>> they're
> > > > > > > > > > >>>>>>>> likely to
> > > > > > > > > > >>>>>>>> be looking at all the others. Then naming this
> one
> > > > > "-mean"
> > > > > > > > would
> > > > > > > > > > >>>> probably
> > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in
> the
> > > > > other
> > > > > > > > metrics
> > > > > > > > > > >>> has
> > > > > > > > > > >>>> a
> > > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and
> > > p75)
> > > > > > > better.
> > > > > > > > I'll
> > > > > > > > > > >>>> swap
> > > > > > > > > > >>>>>>>> that out
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > > > > > vvcephei@apache.org
> > > > > > > > > > >>>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a
> > > > reason
> > > > > to
> > > > > > > > choose
> > > > > > > > > > >>>> "avg"
> > > > > > > > > > >>>>>>>>> instead
> > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I
> might
> > > be
> > > > > > > > > > >> oversensitive
> > > > > > > > > > >>>>>>> because
> > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but it
> strikes
> > > me
> > > > > > that
> > > > > > > > "avg"
> > > > > > > > > > >> is
> > > > > > > > > > >>>>>>>>> actually
> > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of
> statistics,
> > > > > > whereas
> > > > > > > > "mean"
> > > > > > > > > > >>> is
> > > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none
> with
> > > > > > "mean";
> > > > > > > > was
> > > > > > > > > > >>> that
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Regarding the names of the percentile, I
> actually
> > > > > > couldn't
> > > > > > > > find
> > > > > > > > > > >>> _any_
> > > > > > > > > > >>>>>>> other
> > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there a
> reason to
> > > > > choose
> > > > > > > > "99th"
> > > > > > > > > > >> as
> > > > > > > > > > >>>>>>> opposed
> > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> > > > criticism,
> > > > > > I'm
> > > > > > > > just
> > > > > > > > > > >>>>>>> primarily
> > > > > > > > > > >>>>>>>>> asking
> > > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > >>>>>>>>> -John
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie
> Blee-Goldman
> > > > > > wrote:
> > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min
> metric
> > > for
> > > > > the
> > > > > > > > sake of
> > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or
> 50th
> > > > > > > > percentile)
> > > > > > > > > > >>> but I
> > > > > > > > > > >>>>>>> can
> > > > > > > > > > >>>>>>>>> see
> > > > > > > > > > >>>>>>>>>> how users might expect it and find it a bit
> > > > > disorienting
> > > > > > > > not to
> > > > > > > > > > >>>>>>> have. So
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > > percentile)*
> > > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > > percentile)*
> > > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to
> embarrass
> > > > > > myself.
> > > > > > > I
> > > > > > > > > > >> guess
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> mean
> > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the
> median, but
> > > > > > neither
> > > > > > > > are
> > > > > > > > > > >> as
> > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th
> and
> > > > max).
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but
> it
> > > > does
> > > > > > > round
> > > > > > > > out
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>>>> mental
> > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always
> graphed
> > > > min
> > > > > > > along
> > > > > > > > with
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > >>>>>>>>>>> metrics to help me understand how fast the
> system
> > > > can
> > > > > > be,
> > > > > > > > which
> > > > > > > > > > >>>>>>> helps
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a
> relatively
> > > > > > > inexpensive
> > > > > > > > > > >> metric
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just throw
> it in.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> > > Blee-Goldman
> > > > > > > wrote:
> > > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end
> latency
> > > > *up*
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >>>>>>> specific
> > > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that
> drives
> > > > the
> > > > > > > point
> > > > > > > > > > >> home
> > > > > > > > > > >>>>>>>>> better.
> > > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is
> received by
> > > > that
> > > > > > > > task" --
> > > > > > > > > > >>>>>>> will
> > > > > > > > > > >>>>>>>>> update
> > > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the
> min
> > > > would
> > > > > > > > actually
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't
> feel
> > > > > strongly
> > > > > > > > that we
> > > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can
> you
> > > > > > > enlighten
> > > > > > > > me
> > > > > > > > > > >>>>>>> with an
> > > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the
> overhead
> > > of
> > > > > > > adding
> > > > > > > > > > >>>>>>> multiple
> > > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate
> what
> > > kind
> > > > > of
> > > > > > > > > > >>> performance
> > > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th
> > > > percentile
> > > > > > > > would be
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>>>> median
> > > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean
> *and*
> > > the
> > > > > 50th
> > > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John
> Roesler <
> > > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so
> far:
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether
> we
> > > > report
> > > > > > > > "spans"
> > > > > > > > > > >> or
> > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice,
> some of
> > > > the
> > > > > > > spans
> > > > > > > > are
> > > > > > > > > > >>>>>>> pretty
> > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like time
> spent
> > > > > waiting
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > > >>>>>>>>>>>>> time from the event happening to the ETL
> > > producer
> > > > > > > > choosing to
> > > > > > > > > > >>>>>>> send
> > > > > > > > > > >>>>>>>>> it,
> > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers,
> etc.,
> > > etc.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically easier to
> > > > compute
> > > > > > > spans
> > > > > > > > by
> > > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e
> > > latencies
> > > > > by
> > > > > > > > adding
> > > > > > > > > > >>>>>>> spans.
> > > > > > > > > > >>>>>>>>> You
> > > > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> > > computation
> > > > > from
> > > > > > > e2e
> > > > > > > > > > >>>>>>> always
> > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas
> > > > computing
> > > > > > e2e
> > > > > > > > > > >> latency
> > > > > > > > > > >>>>>>>>> from
> > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans
> leading
> > > up
> > > > to
> > > > > > the
> > > > > > > > end
> > > > > > > > > > >> you
> > > > > > > > > > >>>>>>>>> care
> > > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to have
> > > spans
> > > > > when
> > > > > > > > they
> > > > > > > > > > >> are
> > > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a
> more
> > > > > > general
> > > > > > > > > > >>>>>>> measurement
> > > > > > > > > > >>>>>>>>>>>>> that basically every person/application
> cares
> > > > about
> > > > > > and
> > > > > > > > > > >> should
> > > > > > > > > > >>>>>>> be
> > > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more
> > > value
> > > > to
> > > > > > > more
> > > > > > > > > > >>>>>>> people if
> > > > > > > > > > >>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding
> "record-staleness" as
> > > a
> > > > > > name,
> > > > > > > I
> > > > > > > > > > >> think
> > > > > > > > > > >>>>>>> I
> > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> > > > intuition.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice,
> > > since
> > > > > the
> > > > > > > > inside
> > > > > > > > > > >> of
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>>>> task
> > > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of latency
> in
> > > some
> > > > > > > cases.
> > > > > > > > > > >> Plus,
> > > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to
> > > know
> > > > > (for
> > > > > > > the
> > > > > > > > > > >>>>>>> purposes
> > > > > > > > > > >>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes source
> events
> > > to
> > > > > > "show
> > > > > > > > up" at
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at every
> > > processor
> > > > > > could
> > > > > > > be
> > > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the
> node
> > > > > level.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record
> also.
> > > > > Even
> > > > > > if
> > > > > > > > it's
> > > > > > > > > > >> a
> > > > > > > > > > >>>>>>> poor
> > > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people
> might
> > > > > > expect
> > > > > > > > to see
> > > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there
> a
> > > > reason
> > > > > > to
> > > > > > > > leave
> > > > > > > > > > >> it
> > > > > > > > > > >>>>>>>>> off?
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > > >>>>>>>>>>>>> I agree we should include late records
> (though
> > > > not
> > > > > > the
> > > > > > > > ones
> > > > > > > > > > >> we
> > > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> > > > > legitimately
> > > > > > > > some
> > > > > > > > > > >>>>>>> records
> > > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole
> > > point
> > > > > of
> > > > > > > > these
> > > > > > > > > > >>>>>>> metrics.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other
> > > > feedback,
> > > > > > > other
> > > > > > > > > > >> than
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang
> Wang
> > > > > wrote:
> > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick
> > > thoughts:
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both
> the
> > > > > > processing
> > > > > > > > > > >>>>>>> latency
> > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in
> intermediate
> > > > > > topics.
> > > > > > > I
> > > > > > > > have
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the
> > > > latency
> > > > > > > > metric of
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>>>> task A
> > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > > > sub-topology
> > > > > > > up-to
> > > > > > > > but
> > > > > > > > > > >>>>>>> not
> > > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name
> > > > "latency"
> > > > > > > > itself,
> > > > > > > > > > >>>>>>> since
> > > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics
> already
> > > > > which
> > > > > > > are
> > > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc,
> > > while
> > > > > > here
> > > > > > > > we are
> > > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the record
> > > timestamp.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > > > "record-staleness"
> > > > > > > > > > >>>>>>> (note we
> > > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which
> > > case
> > > > > > > > recording at
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>> system-time before we start processing the
> > > > record
> > > > > > > sounds
> > > > > > > > > > >> more
> > > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we
> already
> > > > > have a
> > > > > > > > > > >>>>>>> task-level
> > > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific
> node's
> > > e2e
> > > > > > > > latency is
> > > > > > > > > > >>>>>>>>> similar
> > > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level
> processing
> > > > > > > latency.
> > > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness
> metric is
> > > > > > > > sufficient.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > > > > > Blee-Goldman <
> > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly
> > > > useful
> > > > > > > gauge
> > > > > > > > for
> > > > > > > > > > >>>>>>> this
> > > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an
> > > > *upper
> > > > > > > > *bound on
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose
> percentiles
> > > > > that
> > > > > > > > would
> > > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will
> experience,
> > > > and
> > > > > > > what
> > > > > > > > > > >>>>>>> *close
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific
> > > > numbers
> > > > > > and
> > > > > > > > > > >>>>>>> could be
> > > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing
> > > from
> > > > > > users
> > > > > > > on
> > > > > > > > > > >>>>>>> this.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> > > "hop-to-hop
> > > > > > > > latency" in
> > > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> > > subtracting
> > > > > the
> > > > > > > > > > >>>>>>> previous
> > > > > > > > > > >>>>>>>>> node's
> > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do
> it
> > > > either
> > > > > > way
> > > > > > > > since
> > > > > > > > > > >>>>>>>>> you can
> > > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think
> the
> > > > > > > end-to-end
> > > > > > > > > > >>>>>>> latency
> > > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not
> to
> > > > debug
> > > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > > >>>>>>>>> in the
> > > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it
> takes
> > > > > arecord
> > > > > > to
> > > > > > > > be
> > > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this might
> be
> > > > useful
> > > > > > for
> > > > > > > > users
> > > > > > > > > > >>>>>>>>> who are
> > > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just
> produced
> > > > will
> > > > > > be
> > > > > > > > > > >>>>>>> included
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect --
> but
> > > > > maybe I
> > > > > > > > didn't
> > > > > > > > > > >>>>>>> make
> > > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this
> in
> > > the
> > > > > KIP.
> > > > > > > The
> > > > > > > > > > >>>>>>> short
> > > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I added a
> > > > paragraph
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >>>>>>> end
> > > > > > > > > > >>>>>>>>> of the
> > > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning
> > > here,
> > > > > > please
> > > > > > > > let
> > > > > > > > > > >>>>>>> me
> > > > > > > > > > >>>>>>>>> know
> > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the
> existing
> > > > > metric
> > > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > > Node#process
> > > > > > method
> > > > > > > > to
> > > > > > > > > > >>>>>>> run
> > > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative to the
> > > event
> > > > > > > > timestamp.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is
> > > pretty
> > > > > > > > confusing
> > > > > > > > > > >>>>>>> so
> > > > > > > > > > >>>>>>>>> I've
> > > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to
> "end-to-end-latency"
> > > > > which I
> > > > > > > > feel
> > > > > > > > > > >>>>>>> better
> > > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang
> Chen
> > > <
> > > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the
> E2E
> > > > > latency
> > > > > > > is
> > > > > > > > > > >>>>>>>>> important
> > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> > > > application.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of
> picking
> > > > 75,
> > > > > > 99
> > > > > > > > and
> > > > > > > > > > >>>>>>> max
> > > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > > > > > percentile
> > > > > > > as
> > > > > > > > > > >>>>>>> well
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > > > > > cumulative,
> > > > > > > > I.E
> > > > > > > > > > >>>>>>> if a
> > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) =
> T(B->C) +
> > > > P(B)
> > > > > =
> > > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the
> captured
> > > > > > > latency,
> > > > > > > > > > >>>>>>> and T()
> > > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records
> between
> > > > two
> > > > > > > nodes,
> > > > > > > > > > >>>>>>>>> including
> > > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe
> having
> > > > > T(B->C)
> > > > > > > and
> > > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> > > > > otherwise
> > > > > > > if
> > > > > > > > > > >>>>>>> there
> > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are
> affected in
> > > > the
> > > > > > same
> > > > > > > > > > >>>>>>> time.
> > > > > > > > > > >>>>>>>>> In the
> > > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful
> only
> > > > when
> > > > > > the
> > > > > > > > > > >>>>>>> record
> > > > > > > > > > >>>>>>>>> exits
> > > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this
> > > record
> > > > > > spent
> > > > > > > > > > >>>>>>> inside
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate
> treatment
> > > for
> > > > > > sink
> > > > > > > > > > >>>>>>> nodes and
> > > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count
> the
> > > time
> > > > > > > > receiving
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution
> here,
> > > > > just
> > > > > > > > want to
> > > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late
> arrival
> > > > > records
> > > > > > > as
> > > > > > > > > > >>>>>>> well,
> > > > > > > > > > >>>>>>>>> they
> > > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > > > > > out-of-order
> > > > > > > > > > >>>>>>>>> records are
> > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records.
> Should we
> > > > > also
> > > > > > > > supply
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should
> just
> > > > take
> > > > > > > care
> > > > > > > > of
> > > > > > > > > > >>>>>>> it
> > > > > > > > > > >>>>>>>>> by
> > > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we
> haven't
> > > > > > discussed
> > > > > > > > its
> > > > > > > > > > >>>>>>>>> relation
> > > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics,
> could
> > > you
> > > > > add
> > > > > > > > some
> > > > > > > > > > >>>>>>>>> context
> > > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use
> which`
> > > > > > tutorial
> > > > > > > > for
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > > > > > Blee-Goldman
> > > > > > > <
> > > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on
> KIP-613
> > > > > which
> > > > > > > aims
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>> add
> > > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please
> take a
> > > > look:
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
FWIW, I’d be fine with TRACE, if it makes the processor-level metrics more palatable to everyone.

Ultimately, I think this is just kicking the can down the road, and we should really just refactor our sensors/metrics to be lower overhead. But that’s too much work to consider right before the release so for now, we can kick that can just a little farther.

Thanks for proposing a solution, Guozhang!

-John

On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote:
> Cool, that's good points -- we should not let our current metrics proposal
> to depend on future roadmaps, since they can still be removed later if ever
> becoming not valuable after all.
> 
> On the other hand, what do you think about introducing TRACE level metrics
> recording and make the the processor-node level metrics on TRACE?
> 
> Guozhang
> 
> On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
> 
> > Yeah, I had proposed to replace the "source + sink" task-level metric
> > with "sink only" task-level metrics plus the stateful node-level ones.
> >
> > I completely agree that this proposal should and would look different
> > once we complete those two proposals, ie decouple caching and
> > consolidate suppression. But as it is right now, caching and suppression
> > cause the actual time-of-processing of records to vary greatly throughout
> > a subtopology such that the delta between a source node and an intermediate
> > one is potentially quite large.
> >
> > For this exact reason I would argue that this metric is still valuable at
> > the node-level
> >  even if the "current time" is actually only updated once for a task.
> > Obviously this
> > would not convey any information about the actual processing latency, but
> > as you
> > stated, this is likely to be a small delta on the scale of milliseconds.
> > But the latency
> > due to a suppression can be on the order of minutes, hours, or even days!
> > If
> > there is more than one suppression or caching in a subtopology, the source
> > and sink
> > e2e latency could be wildly misrepresentative of the e2e latency of a node
> > in the middle.
> >
> > Anyways, I removed this last point from the current KIP document because I
> > felt
> > it was ultimately an implementation detail. But the larger point I would
> > argue is that:
> > if we are truly worried about the performance hit of fetching the system
> > time at every
> > node, we should just not fetch the system time at every node instead of
> > dropping the
> > metric altogether. It may be slightly off but only on the order of ms and
> > not on the order
> > of hours, and it's still more useful than no metric at all
> >
> >
> > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com> wrote:
> >
> > > Hi Sophie,
> > >
> > > I think your motivation makes sense, in fact before KIP-444 we used to
> > have
> > > processor-node-level process latency as DEBUG which would holistically
> > > address the motivation but we removed it since it is indeed quite
> > expensive
> > > (I do not have concrete numbers at hand, but basically you'd have to call
> > > system-wall-clock time a couple of time for each record, each processor
> > > node), and in the long run I feel such scenarios would be less common if
> > we
> > > 1) decouple caching from emitting and 2) always put suppression right
> > > before sink only, and that's also why I've proposed to remove them in
> > > KIP-444.
> > >
> > > I missed an update in your wiki that right now you've updated the
> > > task-level metric to be recorded at the sink node, and thought it is
> > still
> > > at the source node, but assuming it is still the case, then practically
> > any
> > > state store's staleness of the task could roughly be measured as that
> > > task-level staleness plus a small delta (usually tens of mills), and the
> > > down stream task B's staleness - up stream task A's staleness would
> > capture
> > > the suppression effect at the end of A plus the intermediate topic
> > > producing -> consuming latency, which usually would not require to be
> > > distinguished.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <sophie@confluent.io
> > >
> > > wrote:
> > >
> > > > Hey Guozhang,
> > > >
> > > > Thanks for the response. I meant to ask this earlier and forgot, but do
> > > you
> > > > have any data or benchmarking results on hand for the performance hit
> > we
> > > > took due to the metrics? Or maybe better yet, on the performance we
> > > gained
> > > > back by dropping some metrics? I think it would help us to better
> > > > understand
> > > > the tradeoffs with some concrete numbers to compare.
> > > >
> > > > I guess I'm inclined to want to include these metrics at the operator
> > > level
> > > > on
> > > > DEBUG since, in my mind, that's exactly what DEBUG is for: performance-
> > > > focused users will presumably run at INFO level and even those who do
> > > > use DEBUG may only do so temporarily, when they actually have something
> > > > to debug.
> > > >
> > > > Of course, I'm not running a Streams app myself so all this is just
> > > > conjecture.
> > > > But it seems reasonable to interpret the two metrics levels as "minimal
> > > > useful
> > > > information" and "all information that provides insight into the
> > system"
> > > >
> > > > If we can agree there, the question becomes "does this provide any
> > > > additional
> > > > insight?" I would still argue that it does. To give a concrete example,
> > > > let's say
> > > > the user has a basic
> > > >
> > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > > >
> > > > subtopology, the application performing IQ on the transformer's state.
> > > The
> > > > user
> > > > wants to know how long it's taking records to be reflected in the
> > results
> > > > of a query.
> > > > With only task-level metrics, they can only guess based on the e2e
> > > latency
> > > > of the
> > > > previous task and of this one. But neither of those is a good
> > > > representation of the
> > > > time it will take a record to be processed by the transformer: the
> > > previous
> > > > task's
> > > > latency will not include the time for the record to go through the
> > > > repartitioning,
> > > > and the current task's latency is measured after a suppression. The
> > > > suppression
> > > > itself will introduce significant latency and doesn't provide an
> > accurate
> > > > estimate.
> > > > Caching would also do what it does best, messing with our understanding
> > > of
> > > > time :)
> > > >
> > > > It also means you can't measure the consumption latency, so there would
> > > be
> > > > no way to tell if the e2e latency is so high because the very first
> > > > subtopology is
> > > > extremely slow, or the record was severely delayed in reaching the
> > input
> > > > topic.
> > > > I guess my earlier proposal to include both source and sink metrics
> > could
> > > > address this issue, but it seems to solve the problem more holistically
> > > --
> > > > and for
> > > > more complex subtopologies -- by including metrics at the processor
> > level
> > > >
> > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hey folks,
> > > > >
> > > > > I want to make a final fight on processor-node-level metrics :) Post
> > > > > KIP-444 we've actually removed a lot of node-level metrics since it
> > is
> > > > too
> > > > > expensive to record at that level and their values are not proven
> > worth
> > > > it.
> > > > > Again I'd use the "we have to either enable all or none of DEBUG
> > > metrics"
> > > > > card, e.g. if we want to look into per-task process-latency which is
> > > > DEBUG,
> > > > > we'd have to record all process-node / state-store / cache level
> > > metrics.
> > > > > On the other hand, within a task we usually do not have a lot of
> > > stateful
> > > > > operator nodes and if we want to find if certain stateful nodes are
> > the
> > > > > bottleneck of process latency, that is usually discoverable from the
> > > > > state-store level read / write latencies already. So I'd imagine that
> > > > > suppose there's a sub-topology with multiple state stores in it, if
> > we
> > > > > found the task-level process latency is high, we can still tell which
> > > > state
> > > > > stores within it is the main bottleneck from the state store read /
> > > write
> > > > > latency (I'm assuming the IO latency is always dominant compared with
> > > > > others).
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > > sophie@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > > @Matthias
> > > > > > > And that's what he says about the 95th percentile! Imagine what
> > he
> > > > > would
> > > > > > say about the 50th :P
> > > > > >
> > > > > > I should have kept watching another 10 minutes. He gets around to
> > > > > covering
> > > > > > the 50th, and let's
> > > > > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > > > > >
> > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks, Sophie!
> > > > > > >
> > > > > > > I think this makes perfect sense. It will be much more intuitive
> > to
> > > > use
> > > > > > > the metric for the stated motivation this way.  I’d be in favor
> > of
> > > > the
> > > > > > > proposal after this update.
> > > > > > >
> > > > > > > Thanks again for taking this on,
> > > > > > > -John
> > > > > > >
> > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to clarify/modify one aspect of this KIP, which is to
> > > > record
> > > > > > the
> > > > > > > > staleness
> > > > > > > > at the *completion* of the record's processing by the
> > > operator/task
> > > > > in
> > > > > > > > question,
> > > > > > > > rather than on the intake. The task-level metrics will be
> > > recorded
> > > > at
> > > > > > the
> > > > > > > > sink
> > > > > > > > node instead of at the source, and the operator-level metrics
> > > will
> > > > be
> > > > > > > > recorded
> > > > > > > > at the end of the operation.
> > > > > > > >
> > > > > > > > The stated purpose and intended usefulness of this KIP is to
> > give
> > > > > > users a
> > > > > > > > way
> > > > > > > > to gauge roughly how long it takes for a record to be reflected
> > > in
> > > > > the
> > > > > > > > "results",
> > > > > > > > whether these results are being read from an output topic or
> > > > through
> > > > > > IQ.
> > > > > > > To
> > > > > > > > take the IQ example, the results of a record are obviously not
> > > > > visible
> > > > > > > until
> > > > > > > > *after* that node has finished processing it. The staleness,
> > > while
> > > > > > still
> > > > > > > > potentially
> > > > > > > > useful as it can impact the *way* a record is processed in a
> > > > stateful
> > > > > > and
> > > > > > > > time-
> > > > > > > > dependent operator, is not part of the problem this KIP
> > > > specifically
> > > > > > set
> > > > > > > out
> > > > > > > > to solve.
> > > > > > > >
> > > > > > > > In light of this I think it's appropriate to revert the name
> > > change
> > > > > > back
> > > > > > > to
> > > > > > > > include
> > > > > > > > latency, since "staleness" as described above only makes sense
> > > when
> > > > > > > > measuring
> > > > > > > > relative to the arrival of a record at a task/node. I'd propose
> > > to
> > > > > save
> > > > > > > the
> > > > > > > > term
> > > > > > > > "staleness" for that particular meaning, and adopt Matthias's
> > > > > > suggestion
> > > > > > > of
> > > > > > > > "record-e2e-latency" for this.
> > > > > > > >
> > > > > > > > Thanks for hanging in there all. Please let me know if you have
> > > any
> > > > > > > > concerns about
> > > > > > > > this change!
> > > > > > > >
> > > > > > > > Sophie
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > > mjsax@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am also happy with max/min/99/90. And I buy your naming
> > > > argument
> > > > > > > about
> > > > > > > > > staleness vs latency.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -Matthias
> > > > > > > > >
> > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > > Hey Sophie,
> > > > > > > > > >
> > > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > > > > sophie@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> @Matthias
> > > > > > > > > >> Regarding tracking the 50th percentile, I'll refer you to
> > > the
> > > > > 4:53
> > > > > > > mark
> > > > > > > > > of
> > > > > > > > > >> the video
> > > > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > > >>
> > > > > > > > > >> And that's what he says about the 95th percentile! Imagine
> > > > what
> > > > > he
> > > > > > > would
> > > > > > > > > >> say about
> > > > > > > > > >> the 50th :P
> > > > > > > > > >>
> > > > > > > > > >> But seriously, since we can't seem to agree that the mean
> > or
> > > > > 50th
> > > > > > > > > >> percentile is actually
> > > > > > > > > >> useful I'm inclined to resurrect my original proposal,
> > > > neither.
> > > > > > But
> > > > > > > I
> > > > > > > > > think
> > > > > > > > > >> that's a good
> > > > > > > > > >> argument against the 75th, which I admittedly chose
> > somewhat
> > > > > > > > > arbitrarily as
> > > > > > > > > >> an
> > > > > > > > > >> intermediate between the 50th and the higher percentiles.
> > > How
> > > > > > about:
> > > > > > > > > >>
> > > > > > > > > >> -max
> > > > > > > > > >> -p99
> > > > > > > > > >> -p90
> > > > > > > > > >> -min
> > > > > > > > > >>
> > > > > > > > > >> with p50/mean still up for debate if anyone feels strongly
> > > for
> > > > > > > either of
> > > > > > > > > >> them.
> > > > > > > > > >>
> > > > > > > > > >> Regarding the name, I've already flip-flopped on this so
> > I'm
> > > > > > > definitely
> > > > > > > > > >> still open to
> > > > > > > > > >> further arguments. But the reason for changing it from
> > > > > > > > > end-to-end-latency
> > > > > > > > > >> (which
> > > > > > > > > >> is similar to what you propose) is that this metric
> > > > technically
> > > > > > > reflects
> > > > > > > > > >> how old (ie how "stale")
> > > > > > > > > >> the record is when it's *received* by the operator, not
> > when
> > > > > it's
> > > > > > > > > processed
> > > > > > > > > >> by the operator.
> > > > > > > > > >> It seemed like there was the potential for confusion that
> > > > > > > > > >> "end-to-end-latency" might
> > > > > > > > > >> represent the latency from the event creation to the time
> > > the
> > > > > > > processor
> > > > > > > > > is
> > > > > > > > > >> done
> > > > > > > > > >> processing it.
> > > > > > > > > >>
> > > > > > > > > >> @John
> > > > > > > > > >> I'd rather err on the side of "not-enough" metrics as we
> > can
> > > > > > always
> > > > > > > add
> > > > > > > > > >> this to the
> > > > > > > > > >> stateless metrics later on. If we decide to measure the
> > time
> > > > at
> > > > > > > every
> > > > > > > > > node
> > > > > > > > > >> and don't
> > > > > > > > > >> find any evidence of a serious performance impact, and
> > users
> > > > > > > indicate
> > > > > > > > > they
> > > > > > > > > >> would
> > > > > > > > > >> like to see this metric at all nodes, then we can easily
> > > start
> > > > > > > reporting
> > > > > > > > > >> them as well.
> > > > > > > > > >> WDYT?
> > > > > > > > > >>
> > > > > > > > > >> That said, sink nodes seem like a reasonable exception to
> > > the
> > > > > > rule.
> > > > > > > > > >> Obviously users
> > > > > > > > > >> should be able to detect the time when the record reaches
> > > the
> > > > > > output
> > > > > > > > > topic
> > > > > > > > > >> but that
> > > > > > > > > >> still leaves a gap in understanding how long the
> > production
> > > > > > latency
> > > > > > > was.
> > > > > > > > > >> This mirrors
> > > > > > > > > >> the consumption latency that is exposed by the task-level
> > > > > metrics,
> > > > > > > which
> > > > > > > > > >> are measured
> > > > > > > > > >> at the source node. For good symmetry what if we actually
> > > > expose
> > > > > > > both
> > > > > > > > > the
> > > > > > > > > >> source
> > > > > > > > > >> and sink latency at the task-level? ie report both sets of
> > > > > > > statistical
> > > > > > > > > >> measurements with
> > > > > > > > > >> the additional tag -source/-sink
> > > > > > > > > >>
> > > > > > > > > >> @Bill
> > > > > > > > > >> Thanks for the comment about regarding the min! I hadn't
> > > > > > considered
> > > > > > > that
> > > > > > > > > >> and it's
> > > > > > > > > >> quite useful to think about how and what is useful from a
> > > > users
> > > > > > > point of
> > > > > > > > > >> view.
> > > > > > > > > >>
> > > > > > > > > >> Regarding your second. point, I'm inclined to leave that
> > as
> > > an
> > > > > > > > > >> implementation detail
> > > > > > > > > >> but my take would be that the user should be allowed to
> > > > control
> > > > > > the
> > > > > > > > > record
> > > > > > > > > >> timestamp
> > > > > > > > > >> used for this with the timestamp extractor. My impression
> > is
> > > > > that
> > > > > > > users
> > > > > > > > > may
> > > > > > > > > >> often embed
> > > > > > > > > >> the actual event time in the payload for whatever reason,
> > > and
> > > > > this
> > > > > > > > > >> represents the "true"
> > > > > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > > bbejeck@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful metric
> > to
> > > > > add.
> > > > > > > > > >>>
> > > > > > > > > >>> Regarding tracking min, I  think it could be valuable for
> > > > users
> > > > > > to
> > > > > > > > > >> discern
> > > > > > > > > >>> which part of their topologies are more efficient since
> > > this
> > > > > is a
> > > > > > > > > >>> task-level metric.  I realize everyone seems to be on
> > board
> > > > > with
> > > > > > > > > >> including
> > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on this topic
> > > > should
> > > > > > we
> > > > > > > > > decide
> > > > > > > > > >>> to revisit adding min or not.
> > > > > > > > > >>>
> > > > > > > > > >>> I do have a question regarding the calculation of
> > > staleness.
> > > > > > > > > >>> Is there going to be a consideration for timestamp
> > > > extractors?
> > > > > > > Users
> > > > > > > > > >> could
> > > > > > > > > >>> prefer to use a timestamp embedded in the payload, and it
> > > > could
> > > > > > > skew
> > > > > > > > > the
> > > > > > > > > >>> measurements.
> > > > > > > > > >>> I was wondering if we should specify in the KIP if
> > setting
> > > > the
> > > > > > > arrival
> > > > > > > > > >> time
> > > > > > > > > >>> is always going to come from the record timestamp, or is
> > > this
> > > > > an
> > > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks!
> > > > > > > > > >>> Bill
> > > > > > > > > >>>
> > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > > > mjsax@apache.org>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think it's not useful to record the avg/mean; it
> > > sensitive
> > > > > to
> > > > > > > > > >>>> outliers. We should rather track the median (50th
> > > > percentile).
> > > > > > > > > >>>>
> > > > > > > > > >>>> Not sure if tracking min is useful, but I am also ok to
> > > > track
> > > > > > it.
> > > > > > > > > >>>>
> > > > > > > > > >>>> However, I find it odd to track 75th percentile.
> > Standard
> > > > > > measures
> > > > > > > > > >> would
> > > > > > > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe
> > > > > picking
> > > > > > > 90th
> > > > > > > > > >>>> might be more useful?
> > > > > > > > > >>>>
> > > > > > > > > >>>> About the name: "staleness" wound really odd, and if
> > fact
> > > > the
> > > > > > > metric
> > > > > > > > > >>>> does capture "latency" so we should call it "latency". I
> > > > > > > understand
> > > > > > > > > the
> > > > > > > > > >>>> issue that we already have a latency metric. So maybe we
> > > > could
> > > > > > > call it
> > > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > > >>>>
> > > > > > > > > >>>> While I agree that we should include out-or-order data
> > > (the
> > > > > KIP
> > > > > > > should
> > > > > > > > > >>>> talk about `out-of-order` data, not `late` data; data is
> > > > only
> > > > > > > `late`
> > > > > > > > > if
> > > > > > > > > >>>> it's out-of-order and if it's dropped), I don't really
> > > > > > understand
> > > > > > > why
> > > > > > > > > >>>> the new metric would help to configure grace period or
> > > > > retention
> > > > > > > time?
> > > > > > > > > >>>> As you mention in the KIP, both are define as max
> > > difference
> > > > > of
> > > > > > > > > >>>> `event-time - stream-time` and thus the new metric that
> > > > takes
> > > > > > > > > >>>> system-/wallclock-time into account does not seem to
> > help
> > > at
> > > > > > all.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> > > > Latency"
> > > > > by
> > > > > > > Gil
> > > > > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> -Matthias
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > > >>>>> Hi Sophie,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> It seems like there would still be plenty of use cases
> > > for
> > > > > > > recording
> > > > > > > > > >>>>> this metric at all processors and not just stateful
> > ones,
> > > > but
> > > > > > I'm
> > > > > > > > > >> happy
> > > > > > > > > >>>>> to suspend my arguments for now. Since you're proposing
> > > to
> > > > > keep
> > > > > > > > > >>>>> them at the processor-node level, it will be seamless
> > > later
> > > > > to
> > > > > > > add
> > > > > > > > > >>>>> in the stateless processors if we want. As a wise man
> > > once
> > > > > > said,
> > > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Regarding the time measurement, it's an implementation
> > > > detail
> > > > > > > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd
> > > > > greatly
> > > > > > > > > >>>>> prefer to measure the system time again when recording
> > > the
> > > > > > > > > >>>>> metric. I don't think we've seen any evidence that
> > proves
> > > > > this
> > > > > > > > > >>>>> would harm performance, and the amount of inaccuracy
> > > using
> > > > > > > > > >>>>> the cached system time could incur is actually
> > > substantial.
> > > > > > But,
> > > > > > > > > >>>>> if you want to just "not mention this" in the KIP, we
> > can
> > > > > defer
> > > > > > > to
> > > > > > > > > >>>>> the actual PR discussion, at which time we're in a
> > better
> > > > > > > position
> > > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Along the lines of the measurement accuracy discussion,
> > > one
> > > > > > > > > >>>>> minor thought I had is that maybe we should consider
> > > > > measuring
> > > > > > > > > >>>>> the task staleness metric at the sink, rather than the
> > > > > source,
> > > > > > so
> > > > > > > > > >> that
> > > > > > > > > >>>>> it includes the processing latency of the task itself,
> > > not
> > > > > just
> > > > > > > the
> > > > > > > > > >>>> latency
> > > > > > > > > >>>>> of everything up to, but not including, the task (which
> > > > seems
> > > > > > > > > >> confusing
> > > > > > > > > >>>>> for users). I guess this could also be an
> > implementation
> > > > > > detail,
> > > > > > > > > >>> though.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thanks for the update,
> > > > > > > > > >>>>> -John
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman
> > > wrote:
> > > > > > > > > >>>>>> Hey all,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> After discussing with Bruno I'd like to propose a
> > small
> > > > > > > amendment,
> > > > > > > > > >>>>>> which is to record the processor-node-level metrics
> > only
> > > > for
> > > > > > > > > >>> *stateful*
> > > > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > > > "processor-node-level"
> > > > > > > > > >>>>>> metric and not a "state-store-level" metric as the
> > > > staleness
> > > > > > is
> > > > > > > > > >> still
> > > > > > > > > >>>>>> a property of the node rather than of the state
> > itself.
> > > > > > > However, it
> > > > > > > > > >>>> seems
> > > > > > > > > >>>>>> that this information is primarily useful for stateful
> > > > > > operators
> > > > > > > > > >> that
> > > > > > > > > >>>> might
> > > > > > > > > >>>>>> be exposing state via IQ or otherwise dependent on the
> > > > > record
> > > > > > > time
> > > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> It's worth calling out that recent performance
> > > > improvements
> > > > > to
> > > > > > > the
> > > > > > > > > >>>> metrics
> > > > > > > > > >>>>>> framework mean that we no longer fetch the system time
> > > at
> > > > > the
> > > > > > > > > >> operator
> > > > > > > > > >>>>>> level, but only once per task. In other words the
> > system
> > > > > time
> > > > > > > is not
> > > > > > > > > >>>> updated
> > > > > > > > > >>>>>> between each process as a record flows through the
> > > > > > subtopology,
> > > > > > > so
> > > > > > > > > >>>>>> debugging the processor-level latency via the
> > > stateleness
> > > > > will
> > > > > > > not
> > > > > > > > > >> be
> > > > > > > > > >>>>>> possible.Note that this doesn't mean the
> > operator-level
> > > > > > metrics
> > > > > > > are
> > > > > > > > > >>> not
> > > > > > > > > >>>>>> *useful* relative to the task-level metric. Upstream
> > > > caching
> > > > > > > and/or
> > > > > > > > > >>>>>> suppression
> > > > > > > > > >>>>>> can still cause a record's staleness at some
> > downstream
> > > > > > stateful
> > > > > > > > > >>>> operator
> > > > > > > > > >>>>>> to deviate from the task-level staleness (recorded at
> > > the
> > > > > > source
> > > > > > > > > >>> node).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Please let me know if you have any concerns about this
> > > > > change.
> > > > > > > The
> > > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > > > > bruno@confluent.io>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > > >>>>>>> I think we do not need it now. If we need it, we can
> > > add
> > > > > it.
> > > > > > > Here
> > > > > > > > > >> the
> > > > > > > > > >>>>>>> old truism applies: Adding is always easier than
> > > > removing.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > > >>>>>>> I think it is good to have the staleness metrics also
> > > on
> > > > > > > > > >>>>>>> processor-node-level. If we do not want to record
> > them
> > > on
> > > > > all
> > > > > > > > > >>>>>>> processor nodes, you could restrict the recording to
> > > > > stateful
> > > > > > > > > >>>>>>> processor-nodes, since those are the ones that would
> > > > > benefit
> > > > > > > most
> > > > > > > > > >>> from
> > > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best,
> > > > > > > > > >>>>>>> Bruno
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Yeah, the specific reason was just to align with the
> > > > > current
> > > > > > > > > >>> metrics.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Is it better to conform than to be right? History
> > has
> > > a
> > > > > lot
> > > > > > > to say
> > > > > > > > > >>> on
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>> matter
> > > > > > > > > >>>>>>>> but I'm not sure how much of it applies to the fine
> > > > > details
> > > > > > of
> > > > > > > > > >>> metrics
> > > > > > > > > >>>>>>>> naming :P
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> More seriously, I figured if people are looking at
> > > this
> > > > > > metric
> > > > > > > > > >>> they're
> > > > > > > > > >>>>>>>> likely to
> > > > > > > > > >>>>>>>> be looking at all the others. Then naming this one
> > > > "-mean"
> > > > > > > would
> > > > > > > > > >>>> probably
> > > > > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the
> > > > other
> > > > > > > metrics
> > > > > > > > > >>> has
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>> different meaning.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and
> > p75)
> > > > > > better.
> > > > > > > I'll
> > > > > > > > > >>>> swap
> > > > > > > > > >>>>>>>> that out
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > > > > vvcephei@apache.org
> > > > > > > > > >>>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a
> > > reason
> > > > to
> > > > > > > choose
> > > > > > > > > >>>> "avg"
> > > > > > > > > >>>>>>>>> instead
> > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might
> > be
> > > > > > > > > >> oversensitive
> > > > > > > > > >>>>>>> because
> > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but it strikes
> > me
> > > > > that
> > > > > > > "avg"
> > > > > > > > > >> is
> > > > > > > > > >>>>>>>>> actually
> > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of statistics,
> > > > > whereas
> > > > > > > "mean"
> > > > > > > > > >>> is
> > > > > > > > > >>>>>>>>> specific.
> > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with
> > > > > "mean";
> > > > > > > was
> > > > > > > > > >>> that
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Regarding the names of the percentile, I actually
> > > > > couldn't
> > > > > > > find
> > > > > > > > > >>> _any_
> > > > > > > > > >>>>>>> other
> > > > > > > > > >>>>>>>>> metrics that use percentile. Was there a reason to
> > > > choose
> > > > > > > "99th"
> > > > > > > > > >> as
> > > > > > > > > >>>>>>> opposed
> > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> > > criticism,
> > > > > I'm
> > > > > > > just
> > > > > > > > > >>>>>>> primarily
> > > > > > > > > >>>>>>>>> asking
> > > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > >>>>>>>>> -John
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman
> > > > > wrote:
> > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min metric
> > for
> > > > the
> > > > > > > sake of
> > > > > > > > > >>>>>>> pretty
> > > > > > > > > >>>>>>>>>> graphs
> > > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > > > > > > percentile)
> > > > > > > > > >>> but I
> > > > > > > > > >>>>>>> can
> > > > > > > > > >>>>>>>>> see
> > > > > > > > > >>>>>>>>>> how users might expect it and find it a bit
> > > > disorienting
> > > > > > > not to
> > > > > > > > > >>>>>>> have. So
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> > percentile)*
> > > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> > percentile)*
> > > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > > > > >> vvcephei@apache.org
> > > > > > > > > >>>>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass
> > > > > myself.
> > > > > > I
> > > > > > > > > >> guess
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> mean
> > > > > > > > > >>>>>>>>>>> seems more interesting to me than the median, but
> > > > > neither
> > > > > > > are
> > > > > > > > > >> as
> > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th and
> > > max).
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it
> > > does
> > > > > > round
> > > > > > > out
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>>>> mental
> > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed
> > > min
> > > > > > along
> > > > > > > with
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>> metrics to help me understand how fast the system
> > > can
> > > > > be,
> > > > > > > which
> > > > > > > > > >>>>>>> helps
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> > > > > > inexpensive
> > > > > > > > > >> metric
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> > Blee-Goldman
> > > > > > wrote:
> > > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency
> > > *up*
> > > > > to
> > > > > > > the
> > > > > > > > > >>>>>>> specific
> > > > > > > > > >>>>>>>>>>> task"
> > > > > > > > > >>>>>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives
> > > the
> > > > > > point
> > > > > > > > > >> home
> > > > > > > > > >>>>>>>>> better.
> > > > > > > > > >>>>>>>>>>> So
> > > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is received by
> > > that
> > > > > > > task" --
> > > > > > > > > >>>>>>> will
> > > > > > > > > >>>>>>>>> update
> > > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min
> > > would
> > > > > > > actually
> > > > > > > > > >> be
> > > > > > > > > >>>>>>>>> useful,
> > > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't feel
> > > > strongly
> > > > > > > that we
> > > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> > > > > > enlighten
> > > > > > > me
> > > > > > > > > >>>>>>> with an
> > > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead
> > of
> > > > > > adding
> > > > > > > > > >>>>>>> multiple
> > > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what
> > kind
> > > > of
> > > > > > > > > >>> performance
> > > > > > > > > >>>>>>>>> hit we
> > > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th
> > > percentile
> > > > > > > would be
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>>>> median
> > > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and*
> > the
> > > > 50th
> > > > > > > > > >>>>>>> percentile, or
> > > > > > > > > >>>>>>>>>>> just
> > > > > > > > > >>>>>>>>>>>> one
> > > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we
> > > report
> > > > > > > "spans"
> > > > > > > > > >> or
> > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of
> > > the
> > > > > > spans
> > > > > > > are
> > > > > > > > > >>>>>>> pretty
> > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like time spent
> > > > waiting
> > > > > in
> > > > > > > the
> > > > > > > > > >>>>>>>>> topics, or
> > > > > > > > > >>>>>>>>>>>>> time from the event happening to the ETL
> > producer
> > > > > > > choosing to
> > > > > > > > > >>>>>>> send
> > > > > > > > > >>>>>>>>> it,
> > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc.,
> > etc.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> In other words, it's practically easier to
> > > compute
> > > > > > spans
> > > > > > > by
> > > > > > > > > >>>>>>>>> subtracting
> > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e
> > latencies
> > > > by
> > > > > > > adding
> > > > > > > > > >>>>>>> spans.
> > > > > > > > > >>>>>>>>> You
> > > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> > computation
> > > > from
> > > > > > e2e
> > > > > > > > > >>>>>>> always
> > > > > > > > > >>>>>>>>> just
> > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas
> > > computing
> > > > > e2e
> > > > > > > > > >> latency
> > > > > > > > > >>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading
> > up
> > > to
> > > > > the
> > > > > > > end
> > > > > > > > > >> you
> > > > > > > > > >>>>>>>>> care
> > > > > > > > > >>>>>>>>>>> about.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to have
> > spans
> > > > when
> > > > > > > they
> > > > > > > > > >> are
> > > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more
> > > > > general
> > > > > > > > > >>>>>>> measurement
> > > > > > > > > >>>>>>>>>>>>> that basically every person/application cares
> > > about
> > > > > and
> > > > > > > > > >> should
> > > > > > > > > >>>>>>> be
> > > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more
> > value
> > > to
> > > > > > more
> > > > > > > > > >>>>>>> people if
> > > > > > > > > >>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as
> > a
> > > > > name,
> > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>> no
> > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> > > intuition.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice,
> > since
> > > > the
> > > > > > > inside
> > > > > > > > > >> of
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>>>> task
> > > > > > > > > >>>>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of latency in
> > some
> > > > > > cases.
> > > > > > > > > >> Plus,
> > > > > > > > > >>>>>>>>> it's a
> > > > > > > > > >>>>>>>>>>> more
> > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to
> > know
> > > > (for
> > > > > > the
> > > > > > > > > >>>>>>> purposes
> > > > > > > > > >>>>>>>>> of
> > > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > > >>>>>>>>>>>>> or something) how long it takes source events
> > to
> > > > > "show
> > > > > > > up" at
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> store.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I think actually recording it at every
> > processor
> > > > > could
> > > > > > be
> > > > > > > > > >>>>>>>>> expensive,
> > > > > > > > > >>>>>>>>>>> but we
> > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the node
> > > > level.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also.
> > > > Even
> > > > > if
> > > > > > > it's
> > > > > > > > > >> a
> > > > > > > > > >>>>>>> poor
> > > > > > > > > >>>>>>>>>>> metric
> > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people might
> > > > > expect
> > > > > > > to see
> > > > > > > > > >>>>>>>>> "mean".
> > > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a
> > > reason
> > > > > to
> > > > > > > leave
> > > > > > > > > >> it
> > > > > > > > > >>>>>>>>> off?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > > >>>>>>>>>>>>> I agree we should include late records (though
> > > not
> > > > > the
> > > > > > > ones
> > > > > > > > > >> we
> > > > > > > > > >>>>>>>>> drop).
> > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> > > > legitimately
> > > > > > > some
> > > > > > > > > >>>>>>> records
> > > > > > > > > >>>>>>>>>>> with a
> > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole
> > point
> > > > of
> > > > > > > these
> > > > > > > > > >>>>>>> metrics.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other
> > > feedback,
> > > > > > other
> > > > > > > > > >> than
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>>>>>> request to
> > > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang
> > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick
> > thoughts:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the
> > > > > processing
> > > > > > > > > >>>>>>> latency
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate
> > > > > topics.
> > > > > > I
> > > > > > > have
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>>>>>> similar
> > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the
> > > latency
> > > > > > > metric of
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>>>> task A
> > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > > sub-topology
> > > > > > up-to
> > > > > > > but
> > > > > > > > > >>>>>>> not
> > > > > > > > > >>>>>>>>>>> including
> > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name
> > > "latency"
> > > > > > > itself,
> > > > > > > > > >>>>>>> since
> > > > > > > > > >>>>>>>>>>> today we
> > > > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics already
> > > > which
> > > > > > are
> > > > > > > > > >>>>>>>>> measuring
> > > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc,
> > while
> > > > > here
> > > > > > > we are
> > > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the record
> > timestamp.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > > "record-staleness"
> > > > > > > > > >>>>>>> (note we
> > > > > > > > > >>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which
> > case
> > > > > > > recording at
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> system-time before we start processing the
> > > record
> > > > > > sounds
> > > > > > > > > >> more
> > > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > > > > >>>>>>> processor-node-level
> > > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we already
> > > > have a
> > > > > > > > > >>>>>>> task-level
> > > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's
> > e2e
> > > > > > > latency is
> > > > > > > > > >>>>>>>>> similar
> > > > > > > > > >>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> > > > > > latency.
> > > > > > > > > >>>>>>>>> Personally I
> > > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > > > > > > sufficient.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > > > > Blee-Goldman <
> > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly
> > > useful
> > > > > > gauge
> > > > > > > for
> > > > > > > > > >>>>>>> this
> > > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an
> > > *upper
> > > > > > > *bound on
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> latency
> > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles
> > > > that
> > > > > > > would
> > > > > > > > > >>>>>>>>> hopefully
> > > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will experience,
> > > and
> > > > > > what
> > > > > > > > > >>>>>>> *close
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>>> all*
> > > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific
> > > numbers
> > > > > and
> > > > > > > > > >>>>>>> could be
> > > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing
> > from
> > > > > users
> > > > > > on
> > > > > > > > > >>>>>>> this.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> > "hop-to-hop
> > > > > > > latency" in
> > > > > > > > > >>>>>>>>> this KIP
> > > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> > subtracting
> > > > the
> > > > > > > > > >>>>>>> previous
> > > > > > > > > >>>>>>>>> node's
> > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it
> > > either
> > > > > way
> > > > > > > since
> > > > > > > > > >>>>>>>>> you can
> > > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> > > > > > end-to-end
> > > > > > > > > >>>>>>> latency
> > > > > > > > > >>>>>>>>>>> feels
> > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to
> > > debug
> > > > > > > > > >>>>>>> bottlenecks
> > > > > > > > > >>>>>>>>> in the
> > > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes
> > > > arecord
> > > > > to
> > > > > > > be
> > > > > > > > > >>>>>>>>> reflected
> > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this might be
> > > useful
> > > > > for
> > > > > > > users
> > > > > > > > > >>>>>>>>> who are
> > > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just produced
> > > will
> > > > > be
> > > > > > > > > >>>>>>> included
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>>> their
> > > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but
> > > > maybe I
> > > > > > > didn't
> > > > > > > > > >>>>>>> make
> > > > > > > > > >>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in
> > the
> > > > KIP.
> > > > > > The
> > > > > > > > > >>>>>>> short
> > > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I added a
> > > paragraph
> > > > > to
> > > > > > > the
> > > > > > > > > >>>>>>> end
> > > > > > > > > >>>>>>>>> of the
> > > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning
> > here,
> > > > > please
> > > > > > > let
> > > > > > > > > >>>>>>> me
> > > > > > > > > >>>>>>>>> know
> > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing
> > > > metric
> > > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> > Node#process
> > > > > method
> > > > > > > to
> > > > > > > > > >>>>>>> run
> > > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative to the
> > event
> > > > > > > timestamp.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is
> > pretty
> > > > > > > confusing
> > > > > > > > > >>>>>>> so
> > > > > > > > > >>>>>>>>> I've
> > > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency"
> > > > which I
> > > > > > > feel
> > > > > > > > > >>>>>>> better
> > > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen
> > <
> > > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E
> > > > latency
> > > > > > is
> > > > > > > > > >>>>>>>>> important
> > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> > > application.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking
> > > 75,
> > > > > 99
> > > > > > > and
> > > > > > > > > >>>>>>> max
> > > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > > > > percentile
> > > > > > as
> > > > > > > > > >>>>>>> well
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > > > > cumulative,
> > > > > > > I.E
> > > > > > > > > >>>>>>> if a
> > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) +
> > > P(B)
> > > > =
> > > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> > > > > > latency,
> > > > > > > > > >>>>>>> and T()
> > > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records between
> > > two
> > > > > > nodes,
> > > > > > > > > >>>>>>>>> including
> > > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having
> > > > T(B->C)
> > > > > > and
> > > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > > >>>>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> > > > otherwise
> > > > > > if
> > > > > > > > > >>>>>>> there
> > > > > > > > > >>>>>>>>> is a
> > > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in
> > > the
> > > > > same
> > > > > > > > > >>>>>>> time.
> > > > > > > > > >>>>>>>>> In the
> > > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only
> > > when
> > > > > the
> > > > > > > > > >>>>>>> record
> > > > > > > > > >>>>>>>>> exits
> > > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this
> > record
> > > > > spent
> > > > > > > > > >>>>>>> inside
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate treatment
> > for
> > > > > sink
> > > > > > > > > >>>>>>> nodes and
> > > > > > > > > >>>>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the
> > time
> > > > > > > receiving
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> record
> > > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here,
> > > > just
> > > > > > > want to
> > > > > > > > > >>>>>>>>> discuss
> > > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival
> > > > records
> > > > > > as
> > > > > > > > > >>>>>>> well,
> > > > > > > > > >>>>>>>>> they
> > > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > > > > out-of-order
> > > > > > > > > >>>>>>>>> records are
> > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we
> > > > also
> > > > > > > supply
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>>>> smooth
> > > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just
> > > take
> > > > > > care
> > > > > > > of
> > > > > > > > > >>>>>>> it
> > > > > > > > > >>>>>>>>> by
> > > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't
> > > > > discussed
> > > > > > > its
> > > > > > > > > >>>>>>>>> relation
> > > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could
> > you
> > > > add
> > > > > > > some
> > > > > > > > > >>>>>>>>> context
> > > > > > > > > >>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which`
> > > > > tutorial
> > > > > > > for
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>> best?
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > > > > Blee-Goldman
> > > > > > <
> > > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613
> > > > which
> > > > > > aims
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>> add
> > > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a
> > > look:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> 
> 
> -- 
> -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
Cool, that's good points -- we should not let our current metrics proposal
to depend on future roadmaps, since they can still be removed later if ever
becoming not valuable after all.

On the other hand, what do you think about introducing TRACE level metrics
recording and make the the processor-node level metrics on TRACE?

Guozhang

On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> Yeah, I had proposed to replace the "source + sink" task-level metric
> with "sink only" task-level metrics plus the stateful node-level ones.
>
> I completely agree that this proposal should and would look different
> once we complete those two proposals, ie decouple caching and
> consolidate suppression. But as it is right now, caching and suppression
> cause the actual time-of-processing of records to vary greatly throughout
> a subtopology such that the delta between a source node and an intermediate
> one is potentially quite large.
>
> For this exact reason I would argue that this metric is still valuable at
> the node-level
>  even if the "current time" is actually only updated once for a task.
> Obviously this
> would not convey any information about the actual processing latency, but
> as you
> stated, this is likely to be a small delta on the scale of milliseconds.
> But the latency
> due to a suppression can be on the order of minutes, hours, or even days!
> If
> there is more than one suppression or caching in a subtopology, the source
> and sink
> e2e latency could be wildly misrepresentative of the e2e latency of a node
> in the middle.
>
> Anyways, I removed this last point from the current KIP document because I
> felt
> it was ultimately an implementation detail. But the larger point I would
> argue is that:
> if we are truly worried about the performance hit of fetching the system
> time at every
> node, we should just not fetch the system time at every node instead of
> dropping the
> metric altogether. It may be slightly off but only on the order of ms and
> not on the order
> of hours, and it's still more useful than no metric at all
>
>
> On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Sophie,
> >
> > I think your motivation makes sense, in fact before KIP-444 we used to
> have
> > processor-node-level process latency as DEBUG which would holistically
> > address the motivation but we removed it since it is indeed quite
> expensive
> > (I do not have concrete numbers at hand, but basically you'd have to call
> > system-wall-clock time a couple of time for each record, each processor
> > node), and in the long run I feel such scenarios would be less common if
> we
> > 1) decouple caching from emitting and 2) always put suppression right
> > before sink only, and that's also why I've proposed to remove them in
> > KIP-444.
> >
> > I missed an update in your wiki that right now you've updated the
> > task-level metric to be recorded at the sink node, and thought it is
> still
> > at the source node, but assuming it is still the case, then practically
> any
> > state store's staleness of the task could roughly be measured as that
> > task-level staleness plus a small delta (usually tens of mills), and the
> > down stream task B's staleness - up stream task A's staleness would
> capture
> > the suppression effect at the end of A plus the intermediate topic
> > producing -> consuming latency, which usually would not require to be
> > distinguished.
> >
> >
> > Guozhang
> >
> >
> > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <sophie@confluent.io
> >
> > wrote:
> >
> > > Hey Guozhang,
> > >
> > > Thanks for the response. I meant to ask this earlier and forgot, but do
> > you
> > > have any data or benchmarking results on hand for the performance hit
> we
> > > took due to the metrics? Or maybe better yet, on the performance we
> > gained
> > > back by dropping some metrics? I think it would help us to better
> > > understand
> > > the tradeoffs with some concrete numbers to compare.
> > >
> > > I guess I'm inclined to want to include these metrics at the operator
> > level
> > > on
> > > DEBUG since, in my mind, that's exactly what DEBUG is for: performance-
> > > focused users will presumably run at INFO level and even those who do
> > > use DEBUG may only do so temporarily, when they actually have something
> > > to debug.
> > >
> > > Of course, I'm not running a Streams app myself so all this is just
> > > conjecture.
> > > But it seems reasonable to interpret the two metrics levels as "minimal
> > > useful
> > > information" and "all information that provides insight into the
> system"
> > >
> > > If we can agree there, the question becomes "does this provide any
> > > additional
> > > insight?" I would still argue that it does. To give a concrete example,
> > > let's say
> > > the user has a basic
> > >
> > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> > >
> > > subtopology, the application performing IQ on the transformer's state.
> > The
> > > user
> > > wants to know how long it's taking records to be reflected in the
> results
> > > of a query.
> > > With only task-level metrics, they can only guess based on the e2e
> > latency
> > > of the
> > > previous task and of this one. But neither of those is a good
> > > representation of the
> > > time it will take a record to be processed by the transformer: the
> > previous
> > > task's
> > > latency will not include the time for the record to go through the
> > > repartitioning,
> > > and the current task's latency is measured after a suppression. The
> > > suppression
> > > itself will introduce significant latency and doesn't provide an
> accurate
> > > estimate.
> > > Caching would also do what it does best, messing with our understanding
> > of
> > > time :)
> > >
> > > It also means you can't measure the consumption latency, so there would
> > be
> > > no way to tell if the e2e latency is so high because the very first
> > > subtopology is
> > > extremely slow, or the record was severely delayed in reaching the
> input
> > > topic.
> > > I guess my earlier proposal to include both source and sink metrics
> could
> > > address this issue, but it seems to solve the problem more holistically
> > --
> > > and for
> > > more complex subtopologies -- by including metrics at the processor
> level
> > >
> > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I want to make a final fight on processor-node-level metrics :) Post
> > > > KIP-444 we've actually removed a lot of node-level metrics since it
> is
> > > too
> > > > expensive to record at that level and their values are not proven
> worth
> > > it.
> > > > Again I'd use the "we have to either enable all or none of DEBUG
> > metrics"
> > > > card, e.g. if we want to look into per-task process-latency which is
> > > DEBUG,
> > > > we'd have to record all process-node / state-store / cache level
> > metrics.
> > > > On the other hand, within a task we usually do not have a lot of
> > stateful
> > > > operator nodes and if we want to find if certain stateful nodes are
> the
> > > > bottleneck of process latency, that is usually discoverable from the
> > > > state-store level read / write latencies already. So I'd imagine that
> > > > suppose there's a sub-topology with multiple state stores in it, if
> we
> > > > found the task-level process latency is high, we can still tell which
> > > state
> > > > stores within it is the main bottleneck from the state store read /
> > write
> > > > latency (I'm assuming the IO latency is always dominant compared with
> > > > others).
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> > sophie@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > > @Matthias
> > > > > > And that's what he says about the 95th percentile! Imagine what
> he
> > > > would
> > > > > say about the 50th :P
> > > > >
> > > > > I should have kept watching another 10 minutes. He gets around to
> > > > covering
> > > > > the 50th, and let's
> > > > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > > > >
> > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Thanks, Sophie!
> > > > > >
> > > > > > I think this makes perfect sense. It will be much more intuitive
> to
> > > use
> > > > > > the metric for the stated motivation this way.  I’d be in favor
> of
> > > the
> > > > > > proposal after this update.
> > > > > >
> > > > > > Thanks again for taking this on,
> > > > > > -John
> > > > > >
> > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to clarify/modify one aspect of this KIP, which is to
> > > record
> > > > > the
> > > > > > > staleness
> > > > > > > at the *completion* of the record's processing by the
> > operator/task
> > > > in
> > > > > > > question,
> > > > > > > rather than on the intake. The task-level metrics will be
> > recorded
> > > at
> > > > > the
> > > > > > > sink
> > > > > > > node instead of at the source, and the operator-level metrics
> > will
> > > be
> > > > > > > recorded
> > > > > > > at the end of the operation.
> > > > > > >
> > > > > > > The stated purpose and intended usefulness of this KIP is to
> give
> > > > > users a
> > > > > > > way
> > > > > > > to gauge roughly how long it takes for a record to be reflected
> > in
> > > > the
> > > > > > > "results",
> > > > > > > whether these results are being read from an output topic or
> > > through
> > > > > IQ.
> > > > > > To
> > > > > > > take the IQ example, the results of a record are obviously not
> > > > visible
> > > > > > until
> > > > > > > *after* that node has finished processing it. The staleness,
> > while
> > > > > still
> > > > > > > potentially
> > > > > > > useful as it can impact the *way* a record is processed in a
> > > stateful
> > > > > and
> > > > > > > time-
> > > > > > > dependent operator, is not part of the problem this KIP
> > > specifically
> > > > > set
> > > > > > out
> > > > > > > to solve.
> > > > > > >
> > > > > > > In light of this I think it's appropriate to revert the name
> > change
> > > > > back
> > > > > > to
> > > > > > > include
> > > > > > > latency, since "staleness" as described above only makes sense
> > when
> > > > > > > measuring
> > > > > > > relative to the arrival of a record at a task/node. I'd propose
> > to
> > > > save
> > > > > > the
> > > > > > > term
> > > > > > > "staleness" for that particular meaning, and adopt Matthias's
> > > > > suggestion
> > > > > > of
> > > > > > > "record-e2e-latency" for this.
> > > > > > >
> > > > > > > Thanks for hanging in there all. Please let me know if you have
> > any
> > > > > > > concerns about
> > > > > > > this change!
> > > > > > >
> > > > > > > Sophie
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> > mjsax@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I am also happy with max/min/99/90. And I buy your naming
> > > argument
> > > > > > about
> > > > > > > > staleness vs latency.
> > > > > > > >
> > > > > > > >
> > > > > > > > -Matthias
> > > > > > > >
> > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > > Hey Sophie,
> > > > > > > > >
> > > > > > > > > 90/99/min/max make sense to me.
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > > > sophie@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> @Matthias
> > > > > > > > >> Regarding tracking the 50th percentile, I'll refer you to
> > the
> > > > 4:53
> > > > > > mark
> > > > > > > > of
> > > > > > > > >> the video
> > > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > > >>
> > > > > > > > >> And that's what he says about the 95th percentile! Imagine
> > > what
> > > > he
> > > > > > would
> > > > > > > > >> say about
> > > > > > > > >> the 50th :P
> > > > > > > > >>
> > > > > > > > >> But seriously, since we can't seem to agree that the mean
> or
> > > > 50th
> > > > > > > > >> percentile is actually
> > > > > > > > >> useful I'm inclined to resurrect my original proposal,
> > > neither.
> > > > > But
> > > > > > I
> > > > > > > > think
> > > > > > > > >> that's a good
> > > > > > > > >> argument against the 75th, which I admittedly chose
> somewhat
> > > > > > > > arbitrarily as
> > > > > > > > >> an
> > > > > > > > >> intermediate between the 50th and the higher percentiles.
> > How
> > > > > about:
> > > > > > > > >>
> > > > > > > > >> -max
> > > > > > > > >> -p99
> > > > > > > > >> -p90
> > > > > > > > >> -min
> > > > > > > > >>
> > > > > > > > >> with p50/mean still up for debate if anyone feels strongly
> > for
> > > > > > either of
> > > > > > > > >> them.
> > > > > > > > >>
> > > > > > > > >> Regarding the name, I've already flip-flopped on this so
> I'm
> > > > > > definitely
> > > > > > > > >> still open to
> > > > > > > > >> further arguments. But the reason for changing it from
> > > > > > > > end-to-end-latency
> > > > > > > > >> (which
> > > > > > > > >> is similar to what you propose) is that this metric
> > > technically
> > > > > > reflects
> > > > > > > > >> how old (ie how "stale")
> > > > > > > > >> the record is when it's *received* by the operator, not
> when
> > > > it's
> > > > > > > > processed
> > > > > > > > >> by the operator.
> > > > > > > > >> It seemed like there was the potential for confusion that
> > > > > > > > >> "end-to-end-latency" might
> > > > > > > > >> represent the latency from the event creation to the time
> > the
> > > > > > processor
> > > > > > > > is
> > > > > > > > >> done
> > > > > > > > >> processing it.
> > > > > > > > >>
> > > > > > > > >> @John
> > > > > > > > >> I'd rather err on the side of "not-enough" metrics as we
> can
> > > > > always
> > > > > > add
> > > > > > > > >> this to the
> > > > > > > > >> stateless metrics later on. If we decide to measure the
> time
> > > at
> > > > > > every
> > > > > > > > node
> > > > > > > > >> and don't
> > > > > > > > >> find any evidence of a serious performance impact, and
> users
> > > > > > indicate
> > > > > > > > they
> > > > > > > > >> would
> > > > > > > > >> like to see this metric at all nodes, then we can easily
> > start
> > > > > > reporting
> > > > > > > > >> them as well.
> > > > > > > > >> WDYT?
> > > > > > > > >>
> > > > > > > > >> That said, sink nodes seem like a reasonable exception to
> > the
> > > > > rule.
> > > > > > > > >> Obviously users
> > > > > > > > >> should be able to detect the time when the record reaches
> > the
> > > > > output
> > > > > > > > topic
> > > > > > > > >> but that
> > > > > > > > >> still leaves a gap in understanding how long the
> production
> > > > > latency
> > > > > > was.
> > > > > > > > >> This mirrors
> > > > > > > > >> the consumption latency that is exposed by the task-level
> > > > metrics,
> > > > > > which
> > > > > > > > >> are measured
> > > > > > > > >> at the source node. For good symmetry what if we actually
> > > expose
> > > > > > both
> > > > > > > > the
> > > > > > > > >> source
> > > > > > > > >> and sink latency at the task-level? ie report both sets of
> > > > > > statistical
> > > > > > > > >> measurements with
> > > > > > > > >> the additional tag -source/-sink
> > > > > > > > >>
> > > > > > > > >> @Bill
> > > > > > > > >> Thanks for the comment about regarding the min! I hadn't
> > > > > considered
> > > > > > that
> > > > > > > > >> and it's
> > > > > > > > >> quite useful to think about how and what is useful from a
> > > users
> > > > > > point of
> > > > > > > > >> view.
> > > > > > > > >>
> > > > > > > > >> Regarding your second. point, I'm inclined to leave that
> as
> > an
> > > > > > > > >> implementation detail
> > > > > > > > >> but my take would be that the user should be allowed to
> > > control
> > > > > the
> > > > > > > > record
> > > > > > > > >> timestamp
> > > > > > > > >> used for this with the timestamp extractor. My impression
> is
> > > > that
> > > > > > users
> > > > > > > > may
> > > > > > > > >> often embed
> > > > > > > > >> the actual event time in the payload for whatever reason,
> > and
> > > > this
> > > > > > > > >> represents the "true"
> > > > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > > bbejeck@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful metric
> to
> > > > add.
> > > > > > > > >>>
> > > > > > > > >>> Regarding tracking min, I  think it could be valuable for
> > > users
> > > > > to
> > > > > > > > >> discern
> > > > > > > > >>> which part of their topologies are more efficient since
> > this
> > > > is a
> > > > > > > > >>> task-level metric.  I realize everyone seems to be on
> board
> > > > with
> > > > > > > > >> including
> > > > > > > > >>> min anyway, but I wanted to add my 2 cents on this topic
> > > should
> > > > > we
> > > > > > > > decide
> > > > > > > > >>> to revisit adding min or not.
> > > > > > > > >>>
> > > > > > > > >>> I do have a question regarding the calculation of
> > staleness.
> > > > > > > > >>> Is there going to be a consideration for timestamp
> > > extractors?
> > > > > > Users
> > > > > > > > >> could
> > > > > > > > >>> prefer to use a timestamp embedded in the payload, and it
> > > could
> > > > > > skew
> > > > > > > > the
> > > > > > > > >>> measurements.
> > > > > > > > >>> I was wondering if we should specify in the KIP if
> setting
> > > the
> > > > > > arrival
> > > > > > > > >> time
> > > > > > > > >>> is always going to come from the record timestamp, or is
> > this
> > > > an
> > > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > > >>>
> > > > > > > > >>> Thanks!
> > > > > > > > >>> Bill
> > > > > > > > >>>
> > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > > mjsax@apache.org>
> > > > > > > > >> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > > >>>>
> > > > > > > > >>>> I think it's not useful to record the avg/mean; it
> > sensitive
> > > > to
> > > > > > > > >>>> outliers. We should rather track the median (50th
> > > percentile).
> > > > > > > > >>>>
> > > > > > > > >>>> Not sure if tracking min is useful, but I am also ok to
> > > track
> > > > > it.
> > > > > > > > >>>>
> > > > > > > > >>>> However, I find it odd to track 75th percentile.
> Standard
> > > > > measures
> > > > > > > > >> would
> > > > > > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe
> > > > picking
> > > > > > 90th
> > > > > > > > >>>> might be more useful?
> > > > > > > > >>>>
> > > > > > > > >>>> About the name: "staleness" wound really odd, and if
> fact
> > > the
> > > > > > metric
> > > > > > > > >>>> does capture "latency" so we should call it "latency". I
> > > > > > understand
> > > > > > > > the
> > > > > > > > >>>> issue that we already have a latency metric. So maybe we
> > > could
> > > > > > call it
> > > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > > >>>>
> > > > > > > > >>>> While I agree that we should include out-or-order data
> > (the
> > > > KIP
> > > > > > should
> > > > > > > > >>>> talk about `out-of-order` data, not `late` data; data is
> > > only
> > > > > > `late`
> > > > > > > > if
> > > > > > > > >>>> it's out-of-order and if it's dropped), I don't really
> > > > > understand
> > > > > > why
> > > > > > > > >>>> the new metric would help to configure grace period or
> > > > retention
> > > > > > time?
> > > > > > > > >>>> As you mention in the KIP, both are define as max
> > difference
> > > > of
> > > > > > > > >>>> `event-time - stream-time` and thus the new metric that
> > > takes
> > > > > > > > >>>> system-/wallclock-time into account does not seem to
> help
> > at
> > > > > all.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> > > Latency"
> > > > by
> > > > > > Gil
> > > > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> -Matthias
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > > >>>>> Hi Sophie,
> > > > > > > > >>>>>
> > > > > > > > >>>>> It seems like there would still be plenty of use cases
> > for
> > > > > > recording
> > > > > > > > >>>>> this metric at all processors and not just stateful
> ones,
> > > but
> > > > > I'm
> > > > > > > > >> happy
> > > > > > > > >>>>> to suspend my arguments for now. Since you're proposing
> > to
> > > > keep
> > > > > > > > >>>>> them at the processor-node level, it will be seamless
> > later
> > > > to
> > > > > > add
> > > > > > > > >>>>> in the stateless processors if we want. As a wise man
> > once
> > > > > said,
> > > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > > >>>>>
> > > > > > > > >>>>> Regarding the time measurement, it's an implementation
> > > detail
> > > > > > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd
> > > > greatly
> > > > > > > > >>>>> prefer to measure the system time again when recording
> > the
> > > > > > > > >>>>> metric. I don't think we've seen any evidence that
> proves
> > > > this
> > > > > > > > >>>>> would harm performance, and the amount of inaccuracy
> > using
> > > > > > > > >>>>> the cached system time could incur is actually
> > substantial.
> > > > > But,
> > > > > > > > >>>>> if you want to just "not mention this" in the KIP, we
> can
> > > > defer
> > > > > > to
> > > > > > > > >>>>> the actual PR discussion, at which time we're in a
> better
> > > > > > position
> > > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Along the lines of the measurement accuracy discussion,
> > one
> > > > > > > > >>>>> minor thought I had is that maybe we should consider
> > > > measuring
> > > > > > > > >>>>> the task staleness metric at the sink, rather than the
> > > > source,
> > > > > so
> > > > > > > > >> that
> > > > > > > > >>>>> it includes the processing latency of the task itself,
> > not
> > > > just
> > > > > > the
> > > > > > > > >>>> latency
> > > > > > > > >>>>> of everything up to, but not including, the task (which
> > > seems
> > > > > > > > >> confusing
> > > > > > > > >>>>> for users). I guess this could also be an
> implementation
> > > > > detail,
> > > > > > > > >>> though.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks for the update,
> > > > > > > > >>>>> -John
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman
> > wrote:
> > > > > > > > >>>>>> Hey all,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> After discussing with Bruno I'd like to propose a
> small
> > > > > > amendment,
> > > > > > > > >>>>>> which is to record the processor-node-level metrics
> only
> > > for
> > > > > > > > >>> *stateful*
> > > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > > "processor-node-level"
> > > > > > > > >>>>>> metric and not a "state-store-level" metric as the
> > > staleness
> > > > > is
> > > > > > > > >> still
> > > > > > > > >>>>>> a property of the node rather than of the state
> itself.
> > > > > > However, it
> > > > > > > > >>>> seems
> > > > > > > > >>>>>> that this information is primarily useful for stateful
> > > > > operators
> > > > > > > > >> that
> > > > > > > > >>>> might
> > > > > > > > >>>>>> be exposing state via IQ or otherwise dependent on the
> > > > record
> > > > > > time
> > > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> It's worth calling out that recent performance
> > > improvements
> > > > to
> > > > > > the
> > > > > > > > >>>> metrics
> > > > > > > > >>>>>> framework mean that we no longer fetch the system time
> > at
> > > > the
> > > > > > > > >> operator
> > > > > > > > >>>>>> level, but only once per task. In other words the
> system
> > > > time
> > > > > > is not
> > > > > > > > >>>> updated
> > > > > > > > >>>>>> between each process as a record flows through the
> > > > > subtopology,
> > > > > > so
> > > > > > > > >>>>>> debugging the processor-level latency via the
> > stateleness
> > > > will
> > > > > > not
> > > > > > > > >> be
> > > > > > > > >>>>>> possible.Note that this doesn't mean the
> operator-level
> > > > > metrics
> > > > > > are
> > > > > > > > >>> not
> > > > > > > > >>>>>> *useful* relative to the task-level metric. Upstream
> > > caching
> > > > > > and/or
> > > > > > > > >>>>>> suppression
> > > > > > > > >>>>>> can still cause a record's staleness at some
> downstream
> > > > > stateful
> > > > > > > > >>>> operator
> > > > > > > > >>>>>> to deviate from the task-level staleness (recorded at
> > the
> > > > > source
> > > > > > > > >>> node).
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Please let me know if you have any concerns about this
> > > > change.
> > > > > > The
> > > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > > > bruno@confluent.io>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hi Sophie,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> 50th percentile:
> > > > > > > > >>>>>>> I think we do not need it now. If we need it, we can
> > add
> > > > it.
> > > > > > Here
> > > > > > > > >> the
> > > > > > > > >>>>>>> old truism applies: Adding is always easier than
> > > removing.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > > >>>>>>> I think it is good to have the staleness metrics also
> > on
> > > > > > > > >>>>>>> processor-node-level. If we do not want to record
> them
> > on
> > > > all
> > > > > > > > >>>>>>> processor nodes, you could restrict the recording to
> > > > stateful
> > > > > > > > >>>>>>> processor-nodes, since those are the ones that would
> > > > benefit
> > > > > > most
> > > > > > > > >>> from
> > > > > > > > >>>>>>> the staleness metrics.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Best,
> > > > > > > > >>>>>>> Bruno
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > > > > > >>>> sophie@confluent.io>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Yeah, the specific reason was just to align with the
> > > > current
> > > > > > > > >>> metrics.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Is it better to conform than to be right? History
> has
> > a
> > > > lot
> > > > > > to say
> > > > > > > > >>> on
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>> matter
> > > > > > > > >>>>>>>> but I'm not sure how much of it applies to the fine
> > > > details
> > > > > of
> > > > > > > > >>> metrics
> > > > > > > > >>>>>>>> naming :P
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> More seriously, I figured if people are looking at
> > this
> > > > > metric
> > > > > > > > >>> they're
> > > > > > > > >>>>>>>> likely to
> > > > > > > > >>>>>>>> be looking at all the others. Then naming this one
> > > "-mean"
> > > > > > would
> > > > > > > > >>>> probably
> > > > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the
> > > other
> > > > > > metrics
> > > > > > > > >>> has
> > > > > > > > >>>> a
> > > > > > > > >>>>>>>> different meaning.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and
> p75)
> > > > > better.
> > > > > > I'll
> > > > > > > > >>>> swap
> > > > > > > > >>>>>>>> that out
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > > > vvcephei@apache.org
> > > > > > > > >>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a
> > reason
> > > to
> > > > > > choose
> > > > > > > > >>>> "avg"
> > > > > > > > >>>>>>>>> instead
> > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might
> be
> > > > > > > > >> oversensitive
> > > > > > > > >>>>>>> because
> > > > > > > > >>>>>>>>> of the mistake I just made earlier, but it strikes
> me
> > > > that
> > > > > > "avg"
> > > > > > > > >> is
> > > > > > > > >>>>>>>>> actually
> > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of statistics,
> > > > whereas
> > > > > > "mean"
> > > > > > > > >>> is
> > > > > > > > >>>>>>>>> specific.
> > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with
> > > > "mean";
> > > > > > was
> > > > > > > > >>> that
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Regarding the names of the percentile, I actually
> > > > couldn't
> > > > > > find
> > > > > > > > >>> _any_
> > > > > > > > >>>>>>> other
> > > > > > > > >>>>>>>>> metrics that use percentile. Was there a reason to
> > > choose
> > > > > > "99th"
> > > > > > > > >> as
> > > > > > > > >>>>>>> opposed
> > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> > criticism,
> > > > I'm
> > > > > > just
> > > > > > > > >>>>>>> primarily
> > > > > > > > >>>>>>>>> asking
> > > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > >>>>>>>>> -John
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman
> > > > wrote:
> > > > > > > > >>>>>>>>>> Alright, I can get behind adding the min metric
> for
> > > the
> > > > > > sake of
> > > > > > > > >>>>>>> pretty
> > > > > > > > >>>>>>>>>> graphs
> > > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > > > > > percentile)
> > > > > > > > >>> but I
> > > > > > > > >>>>>>> can
> > > > > > > > >>>>>>>>> see
> > > > > > > > >>>>>>>>>> how users might expect it and find it a bit
> > > disorienting
> > > > > > not to
> > > > > > > > >>>>>>> have. So
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th
> percentile)*
> > > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th
> percentile)*
> > > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > > > >> vvcephei@apache.org
> > > > > > > > >>>>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass
> > > > myself.
> > > > > I
> > > > > > > > >> guess
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> mean
> > > > > > > > >>>>>>>>>>> seems more interesting to me than the median, but
> > > > neither
> > > > > > are
> > > > > > > > >> as
> > > > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th and
> > max).
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it
> > does
> > > > > round
> > > > > > out
> > > > > > > > >>> the
> > > > > > > > >>>>>>>>> mental
> > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed
> > min
> > > > > along
> > > > > > with
> > > > > > > > >>> the
> > > > > > > > >>>>>>>>> other
> > > > > > > > >>>>>>>>>>> metrics to help me understand how fast the system
> > can
> > > > be,
> > > > > > which
> > > > > > > > >>>>>>> helps
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> > > > > inexpensive
> > > > > > > > >> metric
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie
> Blee-Goldman
> > > > > wrote:
> > > > > > > > >>>>>>>>>>>> G1:
> > > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency
> > *up*
> > > > to
> > > > > > the
> > > > > > > > >>>>>>> specific
> > > > > > > > >>>>>>>>>>> task"
> > > > > > > > >>>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives
> > the
> > > > > point
> > > > > > > > >> home
> > > > > > > > >>>>>>>>> better.
> > > > > > > > >>>>>>>>>>> So
> > > > > > > > >>>>>>>>>>>> it's the
> > > > > > > > >>>>>>>>>>>> "staleness of the record when it is received by
> > that
> > > > > > task" --
> > > > > > > > >>>>>>> will
> > > > > > > > >>>>>>>>> update
> > > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min
> > would
> > > > > > actually
> > > > > > > > >> be
> > > > > > > > >>>>>>>>> useful,
> > > > > > > > >>>>>>>>>>>> rather than
> > > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't feel
> > > strongly
> > > > > > that we
> > > > > > > > >>>>>>>>> shouldn't
> > > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> > > > > enlighten
> > > > > > me
> > > > > > > > >>>>>>> with an
> > > > > > > > >>>>>>>>>>>> example?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead
> of
> > > > > adding
> > > > > > > > >>>>>>> multiple
> > > > > > > > >>>>>>>>>>>> percentile
> > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what
> kind
> > > of
> > > > > > > > >>> performance
> > > > > > > > >>>>>>>>> hit we
> > > > > > > > >>>>>>>>>>>> take on
> > > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th
> > percentile
> > > > > > would be
> > > > > > > > >>> the
> > > > > > > > >>>>>>>>> median
> > > > > > > > >>>>>>>>>>>> not the
> > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and*
> the
> > > 50th
> > > > > > > > >>>>>>> percentile, or
> > > > > > > > >>>>>>>>>>> just
> > > > > > > > >>>>>>>>>>>> one
> > > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we
> > report
> > > > > > "spans"
> > > > > > > > >> or
> > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of
> > the
> > > > > spans
> > > > > > are
> > > > > > > > >>>>>>> pretty
> > > > > > > > >>>>>>>>>>>>> difficult to really measure (like time spent
> > > waiting
> > > > in
> > > > > > the
> > > > > > > > >>>>>>>>> topics, or
> > > > > > > > >>>>>>>>>>>>> time from the event happening to the ETL
> producer
> > > > > > choosing to
> > > > > > > > >>>>>>> send
> > > > > > > > >>>>>>>>> it,
> > > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc.,
> etc.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> In other words, it's practically easier to
> > compute
> > > > > spans
> > > > > > by
> > > > > > > > >>>>>>>>> subtracting
> > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e
> latencies
> > > by
> > > > > > adding
> > > > > > > > >>>>>>> spans.
> > > > > > > > >>>>>>>>> You
> > > > > > > > >>>>>>>>>>>>> can even just consider that the span
> computation
> > > from
> > > > > e2e
> > > > > > > > >>>>>>> always
> > > > > > > > >>>>>>>>> just
> > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas
> > computing
> > > > e2e
> > > > > > > > >> latency
> > > > > > > > >>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading
> up
> > to
> > > > the
> > > > > > end
> > > > > > > > >> you
> > > > > > > > >>>>>>>>> care
> > > > > > > > >>>>>>>>>>> about.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> It seems like people really prefer to have
> spans
> > > when
> > > > > > they
> > > > > > > > >> are
> > > > > > > > >>>>>>>>>>> debugging
> > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more
> > > > general
> > > > > > > > >>>>>>> measurement
> > > > > > > > >>>>>>>>>>>>> that basically every person/application cares
> > about
> > > > and
> > > > > > > > >> should
> > > > > > > > >>>>>>> be
> > > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more
> value
> > to
> > > > > more
> > > > > > > > >>>>>>> people if
> > > > > > > > >>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>> report
> > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as
> a
> > > > name,
> > > > > I
> > > > > > > > >> think
> > > > > > > > >>>>>>> I
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>> no
> > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> > intuition.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice,
> since
> > > the
> > > > > > inside
> > > > > > > > >> of
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>>>> task
> > > > > > > > >>>>>>>>>>> can
> > > > > > > > >>>>>>>>>>>>> introduce a significant amount of latency in
> some
> > > > > cases.
> > > > > > > > >> Plus,
> > > > > > > > >>>>>>>>> it's a
> > > > > > > > >>>>>>>>>>> more
> > > > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to
> know
> > > (for
> > > > > the
> > > > > > > > >>>>>>> purposes
> > > > > > > > >>>>>>>>> of
> > > > > > > > >>>>>>>>>>> IQ
> > > > > > > > >>>>>>>>>>>>> or something) how long it takes source events
> to
> > > > "show
> > > > > > up" at
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> store.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I think actually recording it at every
> processor
> > > > could
> > > > > be
> > > > > > > > >>>>>>>>> expensive,
> > > > > > > > >>>>>>>>>>> but we
> > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the node
> > > level.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also.
> > > Even
> > > > if
> > > > > > it's
> > > > > > > > >> a
> > > > > > > > >>>>>>> poor
> > > > > > > > >>>>>>>>>>> metric
> > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people might
> > > > expect
> > > > > > to see
> > > > > > > > >>>>>>>>> "mean".
> > > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a
> > reason
> > > > to
> > > > > > leave
> > > > > > > > >> it
> > > > > > > > >>>>>>>>> off?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > > >>>>>>>>>>>>> I agree we should include late records (though
> > not
> > > > the
> > > > > > ones
> > > > > > > > >> we
> > > > > > > > >>>>>>>>> drop).
> > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> > > legitimately
> > > > > > some
> > > > > > > > >>>>>>> records
> > > > > > > > >>>>>>>>>>> with a
> > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole
> point
> > > of
> > > > > > these
> > > > > > > > >>>>>>> metrics.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other
> > feedback,
> > > > > other
> > > > > > > > >> than
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>>>>>> request to
> > > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>> -John
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick
> thoughts:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the
> > > > processing
> > > > > > > > >>>>>>> latency
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>> task
> > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate
> > > > topics.
> > > > > I
> > > > > > have
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>>>>>> similar
> > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the
> > latency
> > > > > > metric of
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>>>> task A
> > > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> > sub-topology
> > > > > up-to
> > > > > > but
> > > > > > > > >>>>>>> not
> > > > > > > > >>>>>>>>>>> including
> > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name
> > "latency"
> > > > > > itself,
> > > > > > > > >>>>>>> since
> > > > > > > > >>>>>>>>>>> today we
> > > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics already
> > > which
> > > > > are
> > > > > > > > >>>>>>>>> measuring
> > > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc,
> while
> > > > here
> > > > > > we are
> > > > > > > > >>>>>>>>>>> comparing
> > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> system wallclock time with the record
> timestamp.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > > "record-staleness"
> > > > > > > > >>>>>>> (note we
> > > > > > > > >>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which
> case
> > > > > > recording at
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> system-time before we start processing the
> > record
> > > > > sounds
> > > > > > > > >> more
> > > > > > > > >>>>>>>>>>> natural.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > > > >>>>>>> processor-node-level
> > > > > > > > >>>>>>>>> DEBUG
> > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we already
> > > have a
> > > > > > > > >>>>>>> task-level
> > > > > > > > >>>>>>>>>>>>> processing
> > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's
> e2e
> > > > > > latency is
> > > > > > > > >>>>>>>>> similar
> > > > > > > > >>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> > > > > latency.
> > > > > > > > >>>>>>>>> Personally I
> > > > > > > > >>>>>>>>>>>>> think
> > > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > > > > > sufficient.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > > > Blee-Goldman <
> > > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly
> > useful
> > > > > gauge
> > > > > > for
> > > > > > > > >>>>>>> this
> > > > > > > > >>>>>>>>>>>>> specific
> > > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an
> > *upper
> > > > > > *bound on
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> latency
> > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles
> > > that
> > > > > > would
> > > > > > > > >>>>>>>>> hopefully
> > > > > > > > >>>>>>>>>>>>> give a
> > > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will experience,
> > and
> > > > > what
> > > > > > > > >>>>>>> *close
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>>> all*
> > > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific
> > numbers
> > > > and
> > > > > > > > >>>>>>> could be
> > > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing
> from
> > > > users
> > > > > on
> > > > > > > > >>>>>>> this.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the
> "hop-to-hop
> > > > > > latency" in
> > > > > > > > >>>>>>>>> this KIP
> > > > > > > > >>>>>>>>>>>>> since
> > > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by
> subtracting
> > > the
> > > > > > > > >>>>>>> previous
> > > > > > > > >>>>>>>>> node's
> > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it
> > either
> > > > way
> > > > > > since
> > > > > > > > >>>>>>>>> you can
> > > > > > > > >>>>>>>>>>>>> always
> > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> > > > > end-to-end
> > > > > > > > >>>>>>> latency
> > > > > > > > >>>>>>>>>>> feels
> > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to
> > debug
> > > > > > > > >>>>>>> bottlenecks
> > > > > > > > >>>>>>>>> in the
> > > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes
> > > arecord
> > > > to
> > > > > > be
> > > > > > > > >>>>>>>>> reflected
> > > > > > > > >>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this might be
> > useful
> > > > for
> > > > > > users
> > > > > > > > >>>>>>>>> who are
> > > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just produced
> > will
> > > > be
> > > > > > > > >>>>>>> included
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>> their
> > > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but
> > > maybe I
> > > > > > didn't
> > > > > > > > >>>>>>> make
> > > > > > > > >>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>> clear
> > > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in
> the
> > > KIP.
> > > > > The
> > > > > > > > >>>>>>> short
> > > > > > > > >>>>>>>>>>> answer is
> > > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > > >>>>>>>>>>>>>>> we will include late records. I added a
> > paragraph
> > > > to
> > > > > > the
> > > > > > > > >>>>>>> end
> > > > > > > > >>>>>>>>> of the
> > > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning
> here,
> > > > please
> > > > > > let
> > > > > > > > >>>>>>> me
> > > > > > > > >>>>>>>>> know
> > > > > > > > >>>>>>>>>>> if
> > > > > > > > >>>>>>>>>>>>> you
> > > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing
> > > metric
> > > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal
> Node#process
> > > > method
> > > > > > to
> > > > > > > > >>>>>>> run
> > > > > > > > >>>>>>>>>>> whereas
> > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > > >>>>>>>>>>>>>>> would always be measured relative to the
> event
> > > > > > timestamp.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is
> pretty
> > > > > > confusing
> > > > > > > > >>>>>>> so
> > > > > > > > >>>>>>>>> I've
> > > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency"
> > > which I
> > > > > > feel
> > > > > > > > >>>>>>> better
> > > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen
> <
> > > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E
> > > latency
> > > > > is
> > > > > > > > >>>>>>>>> important
> > > > > > > > >>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> > application.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking
> > 75,
> > > > 99
> > > > > > and
> > > > > > > > >>>>>>> max
> > > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > > > percentile
> > > > > as
> > > > > > > > >>>>>>> well
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> production
> > > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > > > cumulative,
> > > > > > I.E
> > > > > > > > >>>>>>> if a
> > > > > > > > >>>>>>>>>>> record
> > > > > > > > >>>>>>>>>>>>> goes
> > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) +
> > P(B)
> > > =
> > > > > > > > >>>>>>> T(B->C) +
> > > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> > > > > latency,
> > > > > > > > >>>>>>> and T()
> > > > > > > > >>>>>>>>>>>>> represents
> > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records between
> > two
> > > > > nodes,
> > > > > > > > >>>>>>>>> including
> > > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having
> > > T(B->C)
> > > > > and
> > > > > > > > >>>>>>>>> T(A->B)
> > > > > > > > >>>>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>> more
> > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> > > otherwise
> > > > > if
> > > > > > > > >>>>>>> there
> > > > > > > > >>>>>>>>> is a
> > > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in
> > the
> > > > same
> > > > > > > > >>>>>>> time.
> > > > > > > > >>>>>>>>> In the
> > > > > > > > >>>>>>>>>>>>> same
> > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only
> > when
> > > > the
> > > > > > > > >>>>>>> record
> > > > > > > > >>>>>>>>> exits
> > > > > > > > >>>>>>>>>>>>> from the
> > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this
> record
> > > > spent
> > > > > > > > >>>>>>> inside
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate treatment
> for
> > > > sink
> > > > > > > > >>>>>>> nodes and
> > > > > > > > >>>>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the
> time
> > > > > > receiving
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> record
> > > > > > > > >>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here,
> > > just
> > > > > > want to
> > > > > > > > >>>>>>>>> discuss
> > > > > > > > >>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival
> > > records
> > > > > as
> > > > > > > > >>>>>>> well,
> > > > > > > > >>>>>>>>> they
> > > > > > > > >>>>>>>>>>>>> would
> > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > > > out-of-order
> > > > > > > > >>>>>>>>> records are
> > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we
> > > also
> > > > > > supply
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>>>> smooth
> > > > > > > > >>>>>>>>>>>>> version
> > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just
> > take
> > > > > care
> > > > > > of
> > > > > > > > >>>>>>> it
> > > > > > > > >>>>>>>>> by
> > > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't
> > > > discussed
> > > > > > its
> > > > > > > > >>>>>>>>> relation
> > > > > > > > >>>>>>>>>>>>> with our
> > > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could
> you
> > > add
> > > > > > some
> > > > > > > > >>>>>>>>> context
> > > > > > > > >>>>>>>>>>> on
> > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which`
> > > > tutorial
> > > > > > for
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>> best?
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > > > Blee-Goldman
> > > > > <
> > > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613
> > > which
> > > > > aims
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>> add
> > > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a
> > look:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> --
> > > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Yeah, I had proposed to replace the "source + sink" task-level metric
with "sink only" task-level metrics plus the stateful node-level ones.

I completely agree that this proposal should and would look different
once we complete those two proposals, ie decouple caching and
consolidate suppression. But as it is right now, caching and suppression
cause the actual time-of-processing of records to vary greatly throughout
a subtopology such that the delta between a source node and an intermediate
one is potentially quite large.

For this exact reason I would argue that this metric is still valuable at
the node-level
 even if the "current time" is actually only updated once for a task.
Obviously this
would not convey any information about the actual processing latency, but
as you
stated, this is likely to be a small delta on the scale of milliseconds.
But the latency
due to a suppression can be on the order of minutes, hours, or even days! If
there is more than one suppression or caching in a subtopology, the source
and sink
e2e latency could be wildly misrepresentative of the e2e latency of a node
in the middle.

Anyways, I removed this last point from the current KIP document because I
felt
it was ultimately an implementation detail. But the larger point I would
argue is that:
if we are truly worried about the performance hit of fetching the system
time at every
node, we should just not fetch the system time at every node instead of
dropping the
metric altogether. It may be slightly off but only on the order of ms and
not on the order
of hours, and it's still more useful than no metric at all


On Fri, May 15, 2020 at 6:40 PM Guozhang Wang <wa...@gmail.com> wrote:

> Hi Sophie,
>
> I think your motivation makes sense, in fact before KIP-444 we used to have
> processor-node-level process latency as DEBUG which would holistically
> address the motivation but we removed it since it is indeed quite expensive
> (I do not have concrete numbers at hand, but basically you'd have to call
> system-wall-clock time a couple of time for each record, each processor
> node), and in the long run I feel such scenarios would be less common if we
> 1) decouple caching from emitting and 2) always put suppression right
> before sink only, and that's also why I've proposed to remove them in
> KIP-444.
>
> I missed an update in your wiki that right now you've updated the
> task-level metric to be recorded at the sink node, and thought it is still
> at the source node, but assuming it is still the case, then practically any
> state store's staleness of the task could roughly be measured as that
> task-level staleness plus a small delta (usually tens of mills), and the
> down stream task B's staleness - up stream task A's staleness would capture
> the suppression effect at the end of A plus the intermediate topic
> producing -> consuming latency, which usually would not require to be
> distinguished.
>
>
> Guozhang
>
>
> On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
>
> > Hey Guozhang,
> >
> > Thanks for the response. I meant to ask this earlier and forgot, but do
> you
> > have any data or benchmarking results on hand for the performance hit we
> > took due to the metrics? Or maybe better yet, on the performance we
> gained
> > back by dropping some metrics? I think it would help us to better
> > understand
> > the tradeoffs with some concrete numbers to compare.
> >
> > I guess I'm inclined to want to include these metrics at the operator
> level
> > on
> > DEBUG since, in my mind, that's exactly what DEBUG is for: performance-
> > focused users will presumably run at INFO level and even those who do
> > use DEBUG may only do so temporarily, when they actually have something
> > to debug.
> >
> > Of course, I'm not running a Streams app myself so all this is just
> > conjecture.
> > But it seems reasonable to interpret the two metrics levels as "minimal
> > useful
> > information" and "all information that provides insight into the system"
> >
> > If we can agree there, the question becomes "does this provide any
> > additional
> > insight?" I would still argue that it does. To give a concrete example,
> > let's say
> > the user has a basic
> >
> > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
> >
> > subtopology, the application performing IQ on the transformer's state.
> The
> > user
> > wants to know how long it's taking records to be reflected in the results
> > of a query.
> > With only task-level metrics, they can only guess based on the e2e
> latency
> > of the
> > previous task and of this one. But neither of those is a good
> > representation of the
> > time it will take a record to be processed by the transformer: the
> previous
> > task's
> > latency will not include the time for the record to go through the
> > repartitioning,
> > and the current task's latency is measured after a suppression. The
> > suppression
> > itself will introduce significant latency and doesn't provide an accurate
> > estimate.
> > Caching would also do what it does best, messing with our understanding
> of
> > time :)
> >
> > It also means you can't measure the consumption latency, so there would
> be
> > no way to tell if the e2e latency is so high because the very first
> > subtopology is
> > extremely slow, or the record was severely delayed in reaching the input
> > topic.
> > I guess my earlier proposal to include both source and sink metrics could
> > address this issue, but it seems to solve the problem more holistically
> --
> > and for
> > more complex subtopologies -- by including metrics at the processor level
> >
> > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hey folks,
> > >
> > > I want to make a final fight on processor-node-level metrics :) Post
> > > KIP-444 we've actually removed a lot of node-level metrics since it is
> > too
> > > expensive to record at that level and their values are not proven worth
> > it.
> > > Again I'd use the "we have to either enable all or none of DEBUG
> metrics"
> > > card, e.g. if we want to look into per-task process-latency which is
> > DEBUG,
> > > we'd have to record all process-node / state-store / cache level
> metrics.
> > > On the other hand, within a task we usually do not have a lot of
> stateful
> > > operator nodes and if we want to find if certain stateful nodes are the
> > > bottleneck of process latency, that is usually discoverable from the
> > > state-store level read / write latencies already. So I'd imagine that
> > > suppose there's a sub-topology with multiple state stores in it, if we
> > > found the task-level process latency is high, we can still tell which
> > state
> > > stores within it is the main bottleneck from the state store read /
> write
> > > latency (I'm assuming the IO latency is always dominant compared with
> > > others).
> > >
> > > Guozhang
> > >
> > >
> > >
> > >
> > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <
> sophie@confluent.io
> > >
> > > wrote:
> > >
> > > > > @Matthias
> > > > > And that's what he says about the 95th percentile! Imagine what he
> > > would
> > > > say about the 50th :P
> > > >
> > > > I should have kept watching another 10 minutes. He gets around to
> > > covering
> > > > the 50th, and let's
> > > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > > >
> > > > On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > > > Thanks, Sophie!
> > > > >
> > > > > I think this makes perfect sense. It will be much more intuitive to
> > use
> > > > > the metric for the stated motivation this way.  I’d be in favor of
> > the
> > > > > proposal after this update.
> > > > >
> > > > > Thanks again for taking this on,
> > > > > -John
> > > > >
> > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to clarify/modify one aspect of this KIP, which is to
> > record
> > > > the
> > > > > > staleness
> > > > > > at the *completion* of the record's processing by the
> operator/task
> > > in
> > > > > > question,
> > > > > > rather than on the intake. The task-level metrics will be
> recorded
> > at
> > > > the
> > > > > > sink
> > > > > > node instead of at the source, and the operator-level metrics
> will
> > be
> > > > > > recorded
> > > > > > at the end of the operation.
> > > > > >
> > > > > > The stated purpose and intended usefulness of this KIP is to give
> > > > users a
> > > > > > way
> > > > > > to gauge roughly how long it takes for a record to be reflected
> in
> > > the
> > > > > > "results",
> > > > > > whether these results are being read from an output topic or
> > through
> > > > IQ.
> > > > > To
> > > > > > take the IQ example, the results of a record are obviously not
> > > visible
> > > > > until
> > > > > > *after* that node has finished processing it. The staleness,
> while
> > > > still
> > > > > > potentially
> > > > > > useful as it can impact the *way* a record is processed in a
> > stateful
> > > > and
> > > > > > time-
> > > > > > dependent operator, is not part of the problem this KIP
> > specifically
> > > > set
> > > > > out
> > > > > > to solve.
> > > > > >
> > > > > > In light of this I think it's appropriate to revert the name
> change
> > > > back
> > > > > to
> > > > > > include
> > > > > > latency, since "staleness" as described above only makes sense
> when
> > > > > > measuring
> > > > > > relative to the arrival of a record at a task/node. I'd propose
> to
> > > save
> > > > > the
> > > > > > term
> > > > > > "staleness" for that particular meaning, and adopt Matthias's
> > > > suggestion
> > > > > of
> > > > > > "record-e2e-latency" for this.
> > > > > >
> > > > > > Thanks for hanging in there all. Please let me know if you have
> any
> > > > > > concerns about
> > > > > > this change!
> > > > > >
> > > > > > Sophie
> > > > > >
> > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <
> mjsax@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > I am also happy with max/min/99/90. And I buy your naming
> > argument
> > > > > about
> > > > > > > staleness vs latency.
> > > > > > >
> > > > > > >
> > > > > > > -Matthias
> > > > > > >
> > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > > Hey Sophie,
> > > > > > > >
> > > > > > > > 90/99/min/max make sense to me.
> > > > > > > >
> > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > > sophie@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> @Matthias
> > > > > > > >> Regarding tracking the 50th percentile, I'll refer you to
> the
> > > 4:53
> > > > > mark
> > > > > > > of
> > > > > > > >> the video
> > > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > > >>
> > > > > > > >> And that's what he says about the 95th percentile! Imagine
> > what
> > > he
> > > > > would
> > > > > > > >> say about
> > > > > > > >> the 50th :P
> > > > > > > >>
> > > > > > > >> But seriously, since we can't seem to agree that the mean or
> > > 50th
> > > > > > > >> percentile is actually
> > > > > > > >> useful I'm inclined to resurrect my original proposal,
> > neither.
> > > > But
> > > > > I
> > > > > > > think
> > > > > > > >> that's a good
> > > > > > > >> argument against the 75th, which I admittedly chose somewhat
> > > > > > > arbitrarily as
> > > > > > > >> an
> > > > > > > >> intermediate between the 50th and the higher percentiles.
> How
> > > > about:
> > > > > > > >>
> > > > > > > >> -max
> > > > > > > >> -p99
> > > > > > > >> -p90
> > > > > > > >> -min
> > > > > > > >>
> > > > > > > >> with p50/mean still up for debate if anyone feels strongly
> for
> > > > > either of
> > > > > > > >> them.
> > > > > > > >>
> > > > > > > >> Regarding the name, I've already flip-flopped on this so I'm
> > > > > definitely
> > > > > > > >> still open to
> > > > > > > >> further arguments. But the reason for changing it from
> > > > > > > end-to-end-latency
> > > > > > > >> (which
> > > > > > > >> is similar to what you propose) is that this metric
> > technically
> > > > > reflects
> > > > > > > >> how old (ie how "stale")
> > > > > > > >> the record is when it's *received* by the operator, not when
> > > it's
> > > > > > > processed
> > > > > > > >> by the operator.
> > > > > > > >> It seemed like there was the potential for confusion that
> > > > > > > >> "end-to-end-latency" might
> > > > > > > >> represent the latency from the event creation to the time
> the
> > > > > processor
> > > > > > > is
> > > > > > > >> done
> > > > > > > >> processing it.
> > > > > > > >>
> > > > > > > >> @John
> > > > > > > >> I'd rather err on the side of "not-enough" metrics as we can
> > > > always
> > > > > add
> > > > > > > >> this to the
> > > > > > > >> stateless metrics later on. If we decide to measure the time
> > at
> > > > > every
> > > > > > > node
> > > > > > > >> and don't
> > > > > > > >> find any evidence of a serious performance impact, and users
> > > > > indicate
> > > > > > > they
> > > > > > > >> would
> > > > > > > >> like to see this metric at all nodes, then we can easily
> start
> > > > > reporting
> > > > > > > >> them as well.
> > > > > > > >> WDYT?
> > > > > > > >>
> > > > > > > >> That said, sink nodes seem like a reasonable exception to
> the
> > > > rule.
> > > > > > > >> Obviously users
> > > > > > > >> should be able to detect the time when the record reaches
> the
> > > > output
> > > > > > > topic
> > > > > > > >> but that
> > > > > > > >> still leaves a gap in understanding how long the production
> > > > latency
> > > > > was.
> > > > > > > >> This mirrors
> > > > > > > >> the consumption latency that is exposed by the task-level
> > > metrics,
> > > > > which
> > > > > > > >> are measured
> > > > > > > >> at the source node. For good symmetry what if we actually
> > expose
> > > > > both
> > > > > > > the
> > > > > > > >> source
> > > > > > > >> and sink latency at the task-level? ie report both sets of
> > > > > statistical
> > > > > > > >> measurements with
> > > > > > > >> the additional tag -source/-sink
> > > > > > > >>
> > > > > > > >> @Bill
> > > > > > > >> Thanks for the comment about regarding the min! I hadn't
> > > > considered
> > > > > that
> > > > > > > >> and it's
> > > > > > > >> quite useful to think about how and what is useful from a
> > users
> > > > > point of
> > > > > > > >> view.
> > > > > > > >>
> > > > > > > >> Regarding your second. point, I'm inclined to leave that as
> an
> > > > > > > >> implementation detail
> > > > > > > >> but my take would be that the user should be allowed to
> > control
> > > > the
> > > > > > > record
> > > > > > > >> timestamp
> > > > > > > >> used for this with the timestamp extractor. My impression is
> > > that
> > > > > users
> > > > > > > may
> > > > > > > >> often embed
> > > > > > > >> the actual event time in the payload for whatever reason,
> and
> > > this
> > > > > > > >> represents the "true"
> > > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> > bbejeck@gmail.com
> > > >
> > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Thanks for the KIP, Sophie, this will be a useful metric to
> > > add.
> > > > > > > >>>
> > > > > > > >>> Regarding tracking min, I  think it could be valuable for
> > users
> > > > to
> > > > > > > >> discern
> > > > > > > >>> which part of their topologies are more efficient since
> this
> > > is a
> > > > > > > >>> task-level metric.  I realize everyone seems to be on board
> > > with
> > > > > > > >> including
> > > > > > > >>> min anyway, but I wanted to add my 2 cents on this topic
> > should
> > > > we
> > > > > > > decide
> > > > > > > >>> to revisit adding min or not.
> > > > > > > >>>
> > > > > > > >>> I do have a question regarding the calculation of
> staleness.
> > > > > > > >>> Is there going to be a consideration for timestamp
> > extractors?
> > > > > Users
> > > > > > > >> could
> > > > > > > >>> prefer to use a timestamp embedded in the payload, and it
> > could
> > > > > skew
> > > > > > > the
> > > > > > > >>> measurements.
> > > > > > > >>> I was wondering if we should specify in the KIP if setting
> > the
> > > > > arrival
> > > > > > > >> time
> > > > > > > >>> is always going to come from the record timestamp, or is
> this
> > > an
> > > > > > > >>> implementation detail we can cover in the PR?
> > > > > > > >>>
> > > > > > > >>> Thanks!
> > > > > > > >>> Bill
> > > > > > > >>>
> > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > > mjsax@apache.org>
> > > > > > > >> wrote:
> > > > > > > >>>
> > > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > > >>>>
> > > > > > > >>>> I think it's not useful to record the avg/mean; it
> sensitive
> > > to
> > > > > > > >>>> outliers. We should rather track the median (50th
> > percentile).
> > > > > > > >>>>
> > > > > > > >>>> Not sure if tracking min is useful, but I am also ok to
> > track
> > > > it.
> > > > > > > >>>>
> > > > > > > >>>> However, I find it odd to track 75th percentile. Standard
> > > > measures
> > > > > > > >> would
> > > > > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe
> > > picking
> > > > > 90th
> > > > > > > >>>> might be more useful?
> > > > > > > >>>>
> > > > > > > >>>> About the name: "staleness" wound really odd, and if fact
> > the
> > > > > metric
> > > > > > > >>>> does capture "latency" so we should call it "latency". I
> > > > > understand
> > > > > > > the
> > > > > > > >>>> issue that we already have a latency metric. So maybe we
> > could
> > > > > call it
> > > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > > >>>>
> > > > > > > >>>> While I agree that we should include out-or-order data
> (the
> > > KIP
> > > > > should
> > > > > > > >>>> talk about `out-of-order` data, not `late` data; data is
> > only
> > > > > `late`
> > > > > > > if
> > > > > > > >>>> it's out-of-order and if it's dropped), I don't really
> > > > understand
> > > > > why
> > > > > > > >>>> the new metric would help to configure grace period or
> > > retention
> > > > > time?
> > > > > > > >>>> As you mention in the KIP, both are define as max
> difference
> > > of
> > > > > > > >>>> `event-time - stream-time` and thus the new metric that
> > takes
> > > > > > > >>>> system-/wallclock-time into account does not seem to help
> at
> > > > all.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> > Latency"
> > > by
> > > > > Gil
> > > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> -Matthias
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > > >>>>> Hi Sophie,
> > > > > > > >>>>>
> > > > > > > >>>>> It seems like there would still be plenty of use cases
> for
> > > > > recording
> > > > > > > >>>>> this metric at all processors and not just stateful ones,
> > but
> > > > I'm
> > > > > > > >> happy
> > > > > > > >>>>> to suspend my arguments for now. Since you're proposing
> to
> > > keep
> > > > > > > >>>>> them at the processor-node level, it will be seamless
> later
> > > to
> > > > > add
> > > > > > > >>>>> in the stateless processors if we want. As a wise man
> once
> > > > said,
> > > > > > > >>>>> "Adding is always easier than removing."
> > > > > > > >>>>>
> > > > > > > >>>>> Regarding the time measurement, it's an implementation
> > detail
> > > > > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd
> > > greatly
> > > > > > > >>>>> prefer to measure the system time again when recording
> the
> > > > > > > >>>>> metric. I don't think we've seen any evidence that proves
> > > this
> > > > > > > >>>>> would harm performance, and the amount of inaccuracy
> using
> > > > > > > >>>>> the cached system time could incur is actually
> substantial.
> > > > But,
> > > > > > > >>>>> if you want to just "not mention this" in the KIP, we can
> > > defer
> > > > > to
> > > > > > > >>>>> the actual PR discussion, at which time we're in a better
> > > > > position
> > > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > > >>>>>
> > > > > > > >>>>> Along the lines of the measurement accuracy discussion,
> one
> > > > > > > >>>>> minor thought I had is that maybe we should consider
> > > measuring
> > > > > > > >>>>> the task staleness metric at the sink, rather than the
> > > source,
> > > > so
> > > > > > > >> that
> > > > > > > >>>>> it includes the processing latency of the task itself,
> not
> > > just
> > > > > the
> > > > > > > >>>> latency
> > > > > > > >>>>> of everything up to, but not including, the task (which
> > seems
> > > > > > > >> confusing
> > > > > > > >>>>> for users). I guess this could also be an implementation
> > > > detail,
> > > > > > > >>> though.
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks for the update,
> > > > > > > >>>>> -John
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman
> wrote:
> > > > > > > >>>>>> Hey all,
> > > > > > > >>>>>>
> > > > > > > >>>>>> After discussing with Bruno I'd like to propose a small
> > > > > amendment,
> > > > > > > >>>>>> which is to record the processor-node-level metrics only
> > for
> > > > > > > >>> *stateful*
> > > > > > > >>>>>> *operators*. They would still be considered a
> > > > > "processor-node-level"
> > > > > > > >>>>>> metric and not a "state-store-level" metric as the
> > staleness
> > > > is
> > > > > > > >> still
> > > > > > > >>>>>> a property of the node rather than of the state itself.
> > > > > However, it
> > > > > > > >>>> seems
> > > > > > > >>>>>> that this information is primarily useful for stateful
> > > > operators
> > > > > > > >> that
> > > > > > > >>>> might
> > > > > > > >>>>>> be exposing state via IQ or otherwise dependent on the
> > > record
> > > > > time
> > > > > > > >>>>>> unlike a stateless operator.
> > > > > > > >>>>>>
> > > > > > > >>>>>> It's worth calling out that recent performance
> > improvements
> > > to
> > > > > the
> > > > > > > >>>> metrics
> > > > > > > >>>>>> framework mean that we no longer fetch the system time
> at
> > > the
> > > > > > > >> operator
> > > > > > > >>>>>> level, but only once per task. In other words the system
> > > time
> > > > > is not
> > > > > > > >>>> updated
> > > > > > > >>>>>> between each process as a record flows through the
> > > > subtopology,
> > > > > so
> > > > > > > >>>>>> debugging the processor-level latency via the
> stateleness
> > > will
> > > > > not
> > > > > > > >> be
> > > > > > > >>>>>> possible.Note that this doesn't mean the operator-level
> > > > metrics
> > > > > are
> > > > > > > >>> not
> > > > > > > >>>>>> *useful* relative to the task-level metric. Upstream
> > caching
> > > > > and/or
> > > > > > > >>>>>> suppression
> > > > > > > >>>>>> can still cause a record's staleness at some downstream
> > > > stateful
> > > > > > > >>>> operator
> > > > > > > >>>>>> to deviate from the task-level staleness (recorded at
> the
> > > > source
> > > > > > > >>> node).
> > > > > > > >>>>>>
> > > > > > > >>>>>> Please let me know if you have any concerns about this
> > > change.
> > > > > The
> > > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > > bruno@confluent.io>
> > > > > > > >>>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hi Sophie,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thank you for the KIP.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> The KIP looks good to me.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> 50th percentile:
> > > > > > > >>>>>>> I think we do not need it now. If we need it, we can
> add
> > > it.
> > > > > Here
> > > > > > > >> the
> > > > > > > >>>>>>> old truism applies: Adding is always easier than
> > removing.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> processor-node-level metrics:
> > > > > > > >>>>>>> I think it is good to have the staleness metrics also
> on
> > > > > > > >>>>>>> processor-node-level. If we do not want to record them
> on
> > > all
> > > > > > > >>>>>>> processor nodes, you could restrict the recording to
> > > stateful
> > > > > > > >>>>>>> processor-nodes, since those are the ones that would
> > > benefit
> > > > > most
> > > > > > > >>> from
> > > > > > > >>>>>>> the staleness metrics.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Best,
> > > > > > > >>>>>>> Bruno
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > > > > >>>> sophie@confluent.io>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Yeah, the specific reason was just to align with the
> > > current
> > > > > > > >>> metrics.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Is it better to conform than to be right? History has
> a
> > > lot
> > > > > to say
> > > > > > > >>> on
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>> matter
> > > > > > > >>>>>>>> but I'm not sure how much of it applies to the fine
> > > details
> > > > of
> > > > > > > >>> metrics
> > > > > > > >>>>>>>> naming :P
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> More seriously, I figured if people are looking at
> this
> > > > metric
> > > > > > > >>> they're
> > > > > > > >>>>>>>> likely to
> > > > > > > >>>>>>>> be looking at all the others. Then naming this one
> > "-mean"
> > > > > would
> > > > > > > >>>> probably
> > > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the
> > other
> > > > > metrics
> > > > > > > >>> has
> > > > > > > >>>> a
> > > > > > > >>>>>>>> different meaning.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and p75)
> > > > better.
> > > > > I'll
> > > > > > > >>>> swap
> > > > > > > >>>>>>>> that out
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > > vvcephei@apache.org
> > > > > > > >>>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a
> reason
> > to
> > > > > choose
> > > > > > > >>>> "avg"
> > > > > > > >>>>>>>>> instead
> > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > > > > > > >> oversensitive
> > > > > > > >>>>>>> because
> > > > > > > >>>>>>>>> of the mistake I just made earlier, but it strikes me
> > > that
> > > > > "avg"
> > > > > > > >> is
> > > > > > > >>>>>>>>> actually
> > > > > > > >>>>>>>>> ambiguous, as it refers to a family of statistics,
> > > whereas
> > > > > "mean"
> > > > > > > >>> is
> > > > > > > >>>>>>>>> specific.
> > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with
> > > "mean";
> > > > > was
> > > > > > > >>> that
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Regarding the names of the percentile, I actually
> > > couldn't
> > > > > find
> > > > > > > >>> _any_
> > > > > > > >>>>>>> other
> > > > > > > >>>>>>>>> metrics that use percentile. Was there a reason to
> > choose
> > > > > "99th"
> > > > > > > >> as
> > > > > > > >>>>>>> opposed
> > > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a
> criticism,
> > > I'm
> > > > > just
> > > > > > > >>>>>>> primarily
> > > > > > > >>>>>>>>> asking
> > > > > > > >>>>>>>>> for consistency's sake.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks again,
> > > > > > > >>>>>>>>> -John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman
> > > wrote:
> > > > > > > >>>>>>>>>> Alright, I can get behind adding the min metric for
> > the
> > > > > sake of
> > > > > > > >>>>>>> pretty
> > > > > > > >>>>>>>>>> graphs
> > > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > > > > percentile)
> > > > > > > >>> but I
> > > > > > > >>>>>>> can
> > > > > > > >>>>>>>>> see
> > > > > > > >>>>>>>>>> how users might expect it and find it a bit
> > disorienting
> > > > > not to
> > > > > > > >>>>>>> have. So
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > > >> vvcephei@apache.org
> > > > > > > >>>>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass
> > > myself.
> > > > I
> > > > > > > >> guess
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> mean
> > > > > > > >>>>>>>>>>> seems more interesting to me than the median, but
> > > neither
> > > > > are
> > > > > > > >> as
> > > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th and
> max).
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it
> does
> > > > round
> > > > > out
> > > > > > > >>> the
> > > > > > > >>>>>>>>> mental
> > > > > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed
> min
> > > > along
> > > > > with
> > > > > > > >>> the
> > > > > > > >>>>>>>>> other
> > > > > > > >>>>>>>>>>> metrics to help me understand how fast the system
> can
> > > be,
> > > > > which
> > > > > > > >>>>>>> helps
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> > > > inexpensive
> > > > > > > >> metric
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman
> > > > wrote:
> > > > > > > >>>>>>>>>>>> G1:
> > > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency
> *up*
> > > to
> > > > > the
> > > > > > > >>>>>>> specific
> > > > > > > >>>>>>>>>>> task"
> > > > > > > >>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives
> the
> > > > point
> > > > > > > >> home
> > > > > > > >>>>>>>>> better.
> > > > > > > >>>>>>>>>>> So
> > > > > > > >>>>>>>>>>>> it's the
> > > > > > > >>>>>>>>>>>> "staleness of the record when it is received by
> that
> > > > > task" --
> > > > > > > >>>>>>> will
> > > > > > > >>>>>>>>> update
> > > > > > > >>>>>>>>>>>> the KIP
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> B1/J:
> > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min
> would
> > > > > actually
> > > > > > > >> be
> > > > > > > >>>>>>>>> useful,
> > > > > > > >>>>>>>>>>>> rather than
> > > > > > > >>>>>>>>>>>> just intellectually interesting. I don't feel
> > strongly
> > > > > that we
> > > > > > > >>>>>>>>> shouldn't
> > > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> > > > enlighten
> > > > > me
> > > > > > > >>>>>>> with an
> > > > > > > >>>>>>>>>>>> example?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead of
> > > > adding
> > > > > > > >>>>>>> multiple
> > > > > > > >>>>>>>>>>>> percentile
> > > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind
> > of
> > > > > > > >>> performance
> > > > > > > >>>>>>>>> hit we
> > > > > > > >>>>>>>>>>>> take on
> > > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th
> percentile
> > > > > would be
> > > > > > > >>> the
> > > > > > > >>>>>>>>> median
> > > > > > > >>>>>>>>>>>> not the
> > > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the
> > 50th
> > > > > > > >>>>>>> percentile, or
> > > > > > > >>>>>>>>>>> just
> > > > > > > >>>>>>>>>>>> one
> > > > > > > >>>>>>>>>>>> of the two?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > > >>>>>>>>>>>> Sophie
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we
> report
> > > > > "spans"
> > > > > > > >> or
> > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of
> the
> > > > spans
> > > > > are
> > > > > > > >>>>>>> pretty
> > > > > > > >>>>>>>>>>>>> difficult to really measure (like time spent
> > waiting
> > > in
> > > > > the
> > > > > > > >>>>>>>>> topics, or
> > > > > > > >>>>>>>>>>>>> time from the event happening to the ETL producer
> > > > > choosing to
> > > > > > > >>>>>>> send
> > > > > > > >>>>>>>>> it,
> > > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> In other words, it's practically easier to
> compute
> > > > spans
> > > > > by
> > > > > > > >>>>>>>>> subtracting
> > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies
> > by
> > > > > adding
> > > > > > > >>>>>>> spans.
> > > > > > > >>>>>>>>> You
> > > > > > > >>>>>>>>>>>>> can even just consider that the span computation
> > from
> > > > e2e
> > > > > > > >>>>>>> always
> > > > > > > >>>>>>>>> just
> > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas
> computing
> > > e2e
> > > > > > > >> latency
> > > > > > > >>>>>>>>> from
> > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up
> to
> > > the
> > > > > end
> > > > > > > >> you
> > > > > > > >>>>>>>>> care
> > > > > > > >>>>>>>>>>> about.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> It seems like people really prefer to have spans
> > when
> > > > > they
> > > > > > > >> are
> > > > > > > >>>>>>>>>>> debugging
> > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more
> > > general
> > > > > > > >>>>>>> measurement
> > > > > > > >>>>>>>>>>>>> that basically every person/application cares
> about
> > > and
> > > > > > > >> should
> > > > > > > >>>>>>> be
> > > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more value
> to
> > > > more
> > > > > > > >>>>>>> people if
> > > > > > > >>>>>>>>> we
> > > > > > > >>>>>>>>>>>>> report
> > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a
> > > name,
> > > > I
> > > > > > > >> think
> > > > > > > >>>>>>> I
> > > > > > > >>>>>>>>> have
> > > > > > > >>>>>>>>>>> no
> > > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples'
> intuition.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> G2:
> > > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice, since
> > the
> > > > > inside
> > > > > > > >> of
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>> task
> > > > > > > >>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>> introduce a significant amount of latency in some
> > > > cases.
> > > > > > > >> Plus,
> > > > > > > >>>>>>>>> it's a
> > > > > > > >>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to know
> > (for
> > > > the
> > > > > > > >>>>>>> purposes
> > > > > > > >>>>>>>>> of
> > > > > > > >>>>>>>>>>> IQ
> > > > > > > >>>>>>>>>>>>> or something) how long it takes source events to
> > > "show
> > > > > up" at
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> store.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I think actually recording it at every processor
> > > could
> > > > be
> > > > > > > >>>>>>>>> expensive,
> > > > > > > >>>>>>>>>>> but we
> > > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the node
> > level.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> B1:
> > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also.
> > Even
> > > if
> > > > > it's
> > > > > > > >> a
> > > > > > > >>>>>>> poor
> > > > > > > >>>>>>>>>>> metric
> > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people might
> > > expect
> > > > > to see
> > > > > > > >>>>>>>>> "mean".
> > > > > > > >>>>>>>>>>>>> Actually,
> > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a
> reason
> > > to
> > > > > leave
> > > > > > > >> it
> > > > > > > >>>>>>>>> off?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> B3:
> > > > > > > >>>>>>>>>>>>> I agree we should include late records (though
> not
> > > the
> > > > > ones
> > > > > > > >> we
> > > > > > > >>>>>>>>> drop).
> > > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> > legitimately
> > > > > some
> > > > > > > >>>>>>> records
> > > > > > > >>>>>>>>>>> with a
> > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole point
> > of
> > > > > these
> > > > > > > >>>>>>> metrics.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other
> feedback,
> > > > other
> > > > > > > >> than
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>>>> request to
> > > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>> -John
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang
> > wrote:
> > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the
> > > processing
> > > > > > > >>>>>>> latency
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>> task
> > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate
> > > topics.
> > > > I
> > > > > have
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>>>> similar
> > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the
> latency
> > > > > metric of
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>> task A
> > > > > > > >>>>>>>>>>>>>> actually measures the latency of the
> sub-topology
> > > > up-to
> > > > > but
> > > > > > > >>>>>>> not
> > > > > > > >>>>>>>>>>> including
> > > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name
> "latency"
> > > > > itself,
> > > > > > > >>>>>>> since
> > > > > > > >>>>>>>>>>> today we
> > > > > > > >>>>>>>>>>>>>> already have several "latency" metrics already
> > which
> > > > are
> > > > > > > >>>>>>>>> measuring
> > > > > > > >>>>>>>>>>>>> elapsed
> > > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc, while
> > > here
> > > > > we are
> > > > > > > >>>>>>>>>>> comparing
> > > > > > > >>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > > "record-staleness"
> > > > > > > >>>>>>> (note we
> > > > > > > >>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case
> > > > > recording at
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>>>> system-time before we start processing the
> record
> > > > sounds
> > > > > > > >> more
> > > > > > > >>>>>>>>>>> natural.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > > >>>>>>> processor-node-level
> > > > > > > >>>>>>>>> DEBUG
> > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we already
> > have a
> > > > > > > >>>>>>> task-level
> > > > > > > >>>>>>>>>>>>> processing
> > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e
> > > > > latency is
> > > > > > > >>>>>>>>> similar
> > > > > > > >>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> > > > latency.
> > > > > > > >>>>>>>>> Personally I
> > > > > > > >>>>>>>>>>>>> think
> > > > > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > > > > sufficient.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > > Blee-Goldman <
> > > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly
> useful
> > > > gauge
> > > > > for
> > > > > > > >>>>>>> this
> > > > > > > >>>>>>>>>>>>> specific
> > > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an
> *upper
> > > > > *bound on
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> latency
> > > > > > > >>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles
> > that
> > > > > would
> > > > > > > >>>>>>>>> hopefully
> > > > > > > >>>>>>>>>>>>> give a
> > > > > > > >>>>>>>>>>>>>>> good
> > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will experience,
> and
> > > > what
> > > > > > > >>>>>>> *close
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>> all*
> > > > > > > >>>>>>>>>>>>>>> records
> > > > > > > >>>>>>>>>>>>>>> will.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific
> numbers
> > > and
> > > > > > > >>>>>>> could be
> > > > > > > >>>>>>>>>>>>> convinced.
> > > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing from
> > > users
> > > > on
> > > > > > > >>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop
> > > > > latency" in
> > > > > > > >>>>>>>>> this KIP
> > > > > > > >>>>>>>>>>>>> since
> > > > > > > >>>>>>>>>>>>>>> users
> > > > > > > >>>>>>>>>>>>>>> can always compute it themselves by subtracting
> > the
> > > > > > > >>>>>>> previous
> > > > > > > >>>>>>>>> node's
> > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it
> either
> > > way
> > > > > since
> > > > > > > >>>>>>>>> you can
> > > > > > > >>>>>>>>>>>>> always
> > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> > > > end-to-end
> > > > > > > >>>>>>> latency
> > > > > > > >>>>>>>>>>> feels
> > > > > > > >>>>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to
> debug
> > > > > > > >>>>>>> bottlenecks
> > > > > > > >>>>>>>>> in the
> > > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes
> > arecord
> > > to
> > > > > be
> > > > > > > >>>>>>>>> reflected
> > > > > > > >>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > > >>>>>>>>>>>>>>> of the topology. For example this might be
> useful
> > > for
> > > > > users
> > > > > > > >>>>>>>>> who are
> > > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > > >>>>>>>>>>>>>>> roughly when a record that was just produced
> will
> > > be
> > > > > > > >>>>>>> included
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>>> their
> > > > > > > >>>>>>>>>>>>> IQ
> > > > > > > >>>>>>>>>>>>>>> results.
> > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but
> > maybe I
> > > > > didn't
> > > > > > > >>>>>>> make
> > > > > > > >>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>> clear
> > > > > > > >>>>>>>>>>>>>>> enough
> > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in the
> > KIP.
> > > > The
> > > > > > > >>>>>>> short
> > > > > > > >>>>>>>>>>> answer is
> > > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > > >>>>>>>>>>>>>>> we will include late records. I added a
> paragraph
> > > to
> > > > > the
> > > > > > > >>>>>>> end
> > > > > > > >>>>>>>>> of the
> > > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning here,
> > > please
> > > > > let
> > > > > > > >>>>>>> me
> > > > > > > >>>>>>>>> know
> > > > > > > >>>>>>>>>>> if
> > > > > > > >>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing
> > metric
> > > > > > > >>>>>>>>>>> "process-latency",
> > > > > > > >>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal Node#process
> > > method
> > > > > to
> > > > > > > >>>>>>> run
> > > > > > > >>>>>>>>>>> whereas
> > > > > > > >>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>> metric
> > > > > > > >>>>>>>>>>>>>>> would always be measured relative to the event
> > > > > timestamp.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is pretty
> > > > > confusing
> > > > > > > >>>>>>> so
> > > > > > > >>>>>>>>> I've
> > > > > > > >>>>>>>>>>>>> renamed
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency"
> > which I
> > > > > feel
> > > > > > > >>>>>>> better
> > > > > > > >>>>>>>>>>>>> reflects
> > > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E
> > latency
> > > > is
> > > > > > > >>>>>>>>> important
> > > > > > > >>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the
> application.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking
> 75,
> > > 99
> > > > > and
> > > > > > > >>>>>>> max
> > > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > > percentile
> > > > as
> > > > > > > >>>>>>> well
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> production
> > > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > > cumulative,
> > > > > I.E
> > > > > > > >>>>>>> if a
> > > > > > > >>>>>>>>>>> record
> > > > > > > >>>>>>>>>>>>> goes
> > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) +
> P(B)
> > =
> > > > > > > >>>>>>> T(B->C) +
> > > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> > > > latency,
> > > > > > > >>>>>>> and T()
> > > > > > > >>>>>>>>>>>>> represents
> > > > > > > >>>>>>>>>>>>>>>> the time for transiting the records between
> two
> > > > nodes,
> > > > > > > >>>>>>>>> including
> > > > > > > >>>>>>>>>>>>>>> processing
> > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having
> > T(B->C)
> > > > and
> > > > > > > >>>>>>>>> T(A->B)
> > > > > > > >>>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> > otherwise
> > > > if
> > > > > > > >>>>>>> there
> > > > > > > >>>>>>>>> is a
> > > > > > > >>>>>>>>>>>>> spike in
> > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in
> the
> > > same
> > > > > > > >>>>>>> time.
> > > > > > > >>>>>>>>> In the
> > > > > > > >>>>>>>>>>>>> same
> > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only
> when
> > > the
> > > > > > > >>>>>>> record
> > > > > > > >>>>>>>>> exits
> > > > > > > >>>>>>>>>>>>> from the
> > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this record
> > > spent
> > > > > > > >>>>>>> inside
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > > >>>>>>>>>>>>>>>> you think we could have separate treatment for
> > > sink
> > > > > > > >>>>>>> nodes and
> > > > > > > >>>>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time
> > > > > receiving
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> record
> > > > > > > >>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here,
> > just
> > > > > want to
> > > > > > > >>>>>>>>> discuss
> > > > > > > >>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival
> > records
> > > > as
> > > > > > > >>>>>>> well,
> > > > > > > >>>>>>>>> they
> > > > > > > >>>>>>>>>>>>> would
> > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > > out-of-order
> > > > > > > >>>>>>>>> records are
> > > > > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we
> > also
> > > > > supply
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>>>> smooth
> > > > > > > >>>>>>>>>>>>> version
> > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just
> take
> > > > care
> > > > > of
> > > > > > > >>>>>>> it
> > > > > > > >>>>>>>>> by
> > > > > > > >>>>>>>>>>>>> themself?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't
> > > discussed
> > > > > its
> > > > > > > >>>>>>>>> relation
> > > > > > > >>>>>>>>>>>>> with our
> > > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could you
> > add
> > > > > some
> > > > > > > >>>>>>>>> context
> > > > > > > >>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which`
> > > tutorial
> > > > > for
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>> best?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > > Blee-Goldman
> > > > <
> > > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613
> > which
> > > > aims
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>> add
> > > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a
> look:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> --
> > > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
Hi Sophie,

I think your motivation makes sense, in fact before KIP-444 we used to have
processor-node-level process latency as DEBUG which would holistically
address the motivation but we removed it since it is indeed quite expensive
(I do not have concrete numbers at hand, but basically you'd have to call
system-wall-clock time a couple of time for each record, each processor
node), and in the long run I feel such scenarios would be less common if we
1) decouple caching from emitting and 2) always put suppression right
before sink only, and that's also why I've proposed to remove them in
KIP-444.

I missed an update in your wiki that right now you've updated the
task-level metric to be recorded at the sink node, and thought it is still
at the source node, but assuming it is still the case, then practically any
state store's staleness of the task could roughly be measured as that
task-level staleness plus a small delta (usually tens of mills), and the
down stream task B's staleness - up stream task A's staleness would capture
the suppression effect at the end of A plus the intermediate topic
producing -> consuming latency, which usually would not require to be
distinguished.


Guozhang


On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> Hey Guozhang,
>
> Thanks for the response. I meant to ask this earlier and forgot, but do you
> have any data or benchmarking results on hand for the performance hit we
> took due to the metrics? Or maybe better yet, on the performance we gained
> back by dropping some metrics? I think it would help us to better
> understand
> the tradeoffs with some concrete numbers to compare.
>
> I guess I'm inclined to want to include these metrics at the operator level
> on
> DEBUG since, in my mind, that's exactly what DEBUG is for: performance-
> focused users will presumably run at INFO level and even those who do
> use DEBUG may only do so temporarily, when they actually have something
> to debug.
>
> Of course, I'm not running a Streams app myself so all this is just
> conjecture.
> But it seems reasonable to interpret the two metrics levels as "minimal
> useful
> information" and "all information that provides insight into the system"
>
> If we can agree there, the question becomes "does this provide any
> additional
> insight?" I would still argue that it does. To give a concrete example,
> let's say
> the user has a basic
>
> SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK
>
> subtopology, the application performing IQ on the transformer's state. The
> user
> wants to know how long it's taking records to be reflected in the results
> of a query.
> With only task-level metrics, they can only guess based on the e2e latency
> of the
> previous task and of this one. But neither of those is a good
> representation of the
> time it will take a record to be processed by the transformer: the previous
> task's
> latency will not include the time for the record to go through the
> repartitioning,
> and the current task's latency is measured after a suppression. The
> suppression
> itself will introduce significant latency and doesn't provide an accurate
> estimate.
> Caching would also do what it does best, messing with our understanding of
> time :)
>
> It also means you can't measure the consumption latency, so there would be
> no way to tell if the e2e latency is so high because the very first
> subtopology is
> extremely slow, or the record was severely delayed in reaching the input
> topic.
> I guess my earlier proposal to include both source and sink metrics could
> address this issue, but it seems to solve the problem more holistically --
> and for
> more complex subtopologies -- by including metrics at the processor level
>
> On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hey folks,
> >
> > I want to make a final fight on processor-node-level metrics :) Post
> > KIP-444 we've actually removed a lot of node-level metrics since it is
> too
> > expensive to record at that level and their values are not proven worth
> it.
> > Again I'd use the "we have to either enable all or none of DEBUG metrics"
> > card, e.g. if we want to look into per-task process-latency which is
> DEBUG,
> > we'd have to record all process-node / state-store / cache level metrics.
> > On the other hand, within a task we usually do not have a lot of stateful
> > operator nodes and if we want to find if certain stateful nodes are the
> > bottleneck of process latency, that is usually discoverable from the
> > state-store level read / write latencies already. So I'd imagine that
> > suppose there's a sub-topology with multiple state stores in it, if we
> > found the task-level process latency is high, we can still tell which
> state
> > stores within it is the main bottleneck from the state store read / write
> > latency (I'm assuming the IO latency is always dominant compared with
> > others).
> >
> > Guozhang
> >
> >
> >
> >
> > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <sophie@confluent.io
> >
> > wrote:
> >
> > > > @Matthias
> > > > And that's what he says about the 95th percentile! Imagine what he
> > would
> > > say about the 50th :P
> > >
> > > I should have kept watching another 10 minutes. He gets around to
> > covering
> > > the 50th, and let's
> > > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> > >
> > > On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > Thanks, Sophie!
> > > >
> > > > I think this makes perfect sense. It will be much more intuitive to
> use
> > > > the metric for the stated motivation this way.  I’d be in favor of
> the
> > > > proposal after this update.
> > > >
> > > > Thanks again for taking this on,
> > > > -John
> > > >
> > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > > Hi all,
> > > > >
> > > > > I'd like to clarify/modify one aspect of this KIP, which is to
> record
> > > the
> > > > > staleness
> > > > > at the *completion* of the record's processing by the operator/task
> > in
> > > > > question,
> > > > > rather than on the intake. The task-level metrics will be recorded
> at
> > > the
> > > > > sink
> > > > > node instead of at the source, and the operator-level metrics will
> be
> > > > > recorded
> > > > > at the end of the operation.
> > > > >
> > > > > The stated purpose and intended usefulness of this KIP is to give
> > > users a
> > > > > way
> > > > > to gauge roughly how long it takes for a record to be reflected in
> > the
> > > > > "results",
> > > > > whether these results are being read from an output topic or
> through
> > > IQ.
> > > > To
> > > > > take the IQ example, the results of a record are obviously not
> > visible
> > > > until
> > > > > *after* that node has finished processing it. The staleness, while
> > > still
> > > > > potentially
> > > > > useful as it can impact the *way* a record is processed in a
> stateful
> > > and
> > > > > time-
> > > > > dependent operator, is not part of the problem this KIP
> specifically
> > > set
> > > > out
> > > > > to solve.
> > > > >
> > > > > In light of this I think it's appropriate to revert the name change
> > > back
> > > > to
> > > > > include
> > > > > latency, since "staleness" as described above only makes sense when
> > > > > measuring
> > > > > relative to the arrival of a record at a task/node. I'd propose to
> > save
> > > > the
> > > > > term
> > > > > "staleness" for that particular meaning, and adopt Matthias's
> > > suggestion
> > > > of
> > > > > "record-e2e-latency" for this.
> > > > >
> > > > > Thanks for hanging in there all. Please let me know if you have any
> > > > > concerns about
> > > > > this change!
> > > > >
> > > > > Sophie
> > > > >
> > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org>
> > > > wrote:
> > > > >
> > > > > > I am also happy with max/min/99/90. And I buy your naming
> argument
> > > > about
> > > > > > staleness vs latency.
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > > Hey Sophie,
> > > > > > >
> > > > > > > 90/99/min/max make sense to me.
> > > > > > >
> > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > > sophie@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> @Matthias
> > > > > > >> Regarding tracking the 50th percentile, I'll refer you to the
> > 4:53
> > > > mark
> > > > > > of
> > > > > > >> the video
> > > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > > >>
> > > > > > >> And that's what he says about the 95th percentile! Imagine
> what
> > he
> > > > would
> > > > > > >> say about
> > > > > > >> the 50th :P
> > > > > > >>
> > > > > > >> But seriously, since we can't seem to agree that the mean or
> > 50th
> > > > > > >> percentile is actually
> > > > > > >> useful I'm inclined to resurrect my original proposal,
> neither.
> > > But
> > > > I
> > > > > > think
> > > > > > >> that's a good
> > > > > > >> argument against the 75th, which I admittedly chose somewhat
> > > > > > arbitrarily as
> > > > > > >> an
> > > > > > >> intermediate between the 50th and the higher percentiles. How
> > > about:
> > > > > > >>
> > > > > > >> -max
> > > > > > >> -p99
> > > > > > >> -p90
> > > > > > >> -min
> > > > > > >>
> > > > > > >> with p50/mean still up for debate if anyone feels strongly for
> > > > either of
> > > > > > >> them.
> > > > > > >>
> > > > > > >> Regarding the name, I've already flip-flopped on this so I'm
> > > > definitely
> > > > > > >> still open to
> > > > > > >> further arguments. But the reason for changing it from
> > > > > > end-to-end-latency
> > > > > > >> (which
> > > > > > >> is similar to what you propose) is that this metric
> technically
> > > > reflects
> > > > > > >> how old (ie how "stale")
> > > > > > >> the record is when it's *received* by the operator, not when
> > it's
> > > > > > processed
> > > > > > >> by the operator.
> > > > > > >> It seemed like there was the potential for confusion that
> > > > > > >> "end-to-end-latency" might
> > > > > > >> represent the latency from the event creation to the time the
> > > > processor
> > > > > > is
> > > > > > >> done
> > > > > > >> processing it.
> > > > > > >>
> > > > > > >> @John
> > > > > > >> I'd rather err on the side of "not-enough" metrics as we can
> > > always
> > > > add
> > > > > > >> this to the
> > > > > > >> stateless metrics later on. If we decide to measure the time
> at
> > > > every
> > > > > > node
> > > > > > >> and don't
> > > > > > >> find any evidence of a serious performance impact, and users
> > > > indicate
> > > > > > they
> > > > > > >> would
> > > > > > >> like to see this metric at all nodes, then we can easily start
> > > > reporting
> > > > > > >> them as well.
> > > > > > >> WDYT?
> > > > > > >>
> > > > > > >> That said, sink nodes seem like a reasonable exception to the
> > > rule.
> > > > > > >> Obviously users
> > > > > > >> should be able to detect the time when the record reaches the
> > > output
> > > > > > topic
> > > > > > >> but that
> > > > > > >> still leaves a gap in understanding how long the production
> > > latency
> > > > was.
> > > > > > >> This mirrors
> > > > > > >> the consumption latency that is exposed by the task-level
> > metrics,
> > > > which
> > > > > > >> are measured
> > > > > > >> at the source node. For good symmetry what if we actually
> expose
> > > > both
> > > > > > the
> > > > > > >> source
> > > > > > >> and sink latency at the task-level? ie report both sets of
> > > > statistical
> > > > > > >> measurements with
> > > > > > >> the additional tag -source/-sink
> > > > > > >>
> > > > > > >> @Bill
> > > > > > >> Thanks for the comment about regarding the min! I hadn't
> > > considered
> > > > that
> > > > > > >> and it's
> > > > > > >> quite useful to think about how and what is useful from a
> users
> > > > point of
> > > > > > >> view.
> > > > > > >>
> > > > > > >> Regarding your second. point, I'm inclined to leave that as an
> > > > > > >> implementation detail
> > > > > > >> but my take would be that the user should be allowed to
> control
> > > the
> > > > > > record
> > > > > > >> timestamp
> > > > > > >> used for this with the timestamp extractor. My impression is
> > that
> > > > users
> > > > > > may
> > > > > > >> often embed
> > > > > > >> the actual event time in the payload for whatever reason, and
> > this
> > > > > > >> represents the "true"
> > > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <
> bbejeck@gmail.com
> > >
> > > > wrote:
> > > > > > >>
> > > > > > >>> Thanks for the KIP, Sophie, this will be a useful metric to
> > add.
> > > > > > >>>
> > > > > > >>> Regarding tracking min, I  think it could be valuable for
> users
> > > to
> > > > > > >> discern
> > > > > > >>> which part of their topologies are more efficient since this
> > is a
> > > > > > >>> task-level metric.  I realize everyone seems to be on board
> > with
> > > > > > >> including
> > > > > > >>> min anyway, but I wanted to add my 2 cents on this topic
> should
> > > we
> > > > > > decide
> > > > > > >>> to revisit adding min or not.
> > > > > > >>>
> > > > > > >>> I do have a question regarding the calculation of staleness.
> > > > > > >>> Is there going to be a consideration for timestamp
> extractors?
> > > > Users
> > > > > > >> could
> > > > > > >>> prefer to use a timestamp embedded in the payload, and it
> could
> > > > skew
> > > > > > the
> > > > > > >>> measurements.
> > > > > > >>> I was wondering if we should specify in the KIP if setting
> the
> > > > arrival
> > > > > > >> time
> > > > > > >>> is always going to come from the record timestamp, or is this
> > an
> > > > > > >>> implementation detail we can cover in the PR?
> > > > > > >>>
> > > > > > >>> Thanks!
> > > > > > >>> Bill
> > > > > > >>>
> > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > > mjsax@apache.org>
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>>> Thanks for the KIP Sophie.
> > > > > > >>>>
> > > > > > >>>> I think it's not useful to record the avg/mean; it sensitive
> > to
> > > > > > >>>> outliers. We should rather track the median (50th
> percentile).
> > > > > > >>>>
> > > > > > >>>> Not sure if tracking min is useful, but I am also ok to
> track
> > > it.
> > > > > > >>>>
> > > > > > >>>> However, I find it odd to track 75th percentile. Standard
> > > measures
> > > > > > >> would
> > > > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe
> > picking
> > > > 90th
> > > > > > >>>> might be more useful?
> > > > > > >>>>
> > > > > > >>>> About the name: "staleness" wound really odd, and if fact
> the
> > > > metric
> > > > > > >>>> does capture "latency" so we should call it "latency". I
> > > > understand
> > > > > > the
> > > > > > >>>> issue that we already have a latency metric. So maybe we
> could
> > > > call it
> > > > > > >>>> `record-e2e-latency-*` ?
> > > > > > >>>>
> > > > > > >>>> While I agree that we should include out-or-order data (the
> > KIP
> > > > should
> > > > > > >>>> talk about `out-of-order` data, not `late` data; data is
> only
> > > > `late`
> > > > > > if
> > > > > > >>>> it's out-of-order and if it's dropped), I don't really
> > > understand
> > > > why
> > > > > > >>>> the new metric would help to configure grace period or
> > retention
> > > > time?
> > > > > > >>>> As you mention in the KIP, both are define as max difference
> > of
> > > > > > >>>> `event-time - stream-time` and thus the new metric that
> takes
> > > > > > >>>> system-/wallclock-time into account does not seem to help at
> > > all.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Btw: there is a great talk about "How NOT to Measure
> Latency"
> > by
> > > > Gil
> > > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> -Matthias
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > > >>>>> Hi Sophie,
> > > > > > >>>>>
> > > > > > >>>>> It seems like there would still be plenty of use cases for
> > > > recording
> > > > > > >>>>> this metric at all processors and not just stateful ones,
> but
> > > I'm
> > > > > > >> happy
> > > > > > >>>>> to suspend my arguments for now. Since you're proposing to
> > keep
> > > > > > >>>>> them at the processor-node level, it will be seamless later
> > to
> > > > add
> > > > > > >>>>> in the stateless processors if we want. As a wise man once
> > > said,
> > > > > > >>>>> "Adding is always easier than removing."
> > > > > > >>>>>
> > > > > > >>>>> Regarding the time measurement, it's an implementation
> detail
> > > > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd
> > greatly
> > > > > > >>>>> prefer to measure the system time again when recording the
> > > > > > >>>>> metric. I don't think we've seen any evidence that proves
> > this
> > > > > > >>>>> would harm performance, and the amount of inaccuracy using
> > > > > > >>>>> the cached system time could incur is actually substantial.
> > > But,
> > > > > > >>>>> if you want to just "not mention this" in the KIP, we can
> > defer
> > > > to
> > > > > > >>>>> the actual PR discussion, at which time we're in a better
> > > > position
> > > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > > >>>>>
> > > > > > >>>>> Along the lines of the measurement accuracy discussion, one
> > > > > > >>>>> minor thought I had is that maybe we should consider
> > measuring
> > > > > > >>>>> the task staleness metric at the sink, rather than the
> > source,
> > > so
> > > > > > >> that
> > > > > > >>>>> it includes the processing latency of the task itself, not
> > just
> > > > the
> > > > > > >>>> latency
> > > > > > >>>>> of everything up to, but not including, the task (which
> seems
> > > > > > >> confusing
> > > > > > >>>>> for users). I guess this could also be an implementation
> > > detail,
> > > > > > >>> though.
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the update,
> > > > > > >>>>> -John
> > > > > > >>>>>
> > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > > > > > >>>>>> Hey all,
> > > > > > >>>>>>
> > > > > > >>>>>> After discussing with Bruno I'd like to propose a small
> > > > amendment,
> > > > > > >>>>>> which is to record the processor-node-level metrics only
> for
> > > > > > >>> *stateful*
> > > > > > >>>>>> *operators*. They would still be considered a
> > > > "processor-node-level"
> > > > > > >>>>>> metric and not a "state-store-level" metric as the
> staleness
> > > is
> > > > > > >> still
> > > > > > >>>>>> a property of the node rather than of the state itself.
> > > > However, it
> > > > > > >>>> seems
> > > > > > >>>>>> that this information is primarily useful for stateful
> > > operators
> > > > > > >> that
> > > > > > >>>> might
> > > > > > >>>>>> be exposing state via IQ or otherwise dependent on the
> > record
> > > > time
> > > > > > >>>>>> unlike a stateless operator.
> > > > > > >>>>>>
> > > > > > >>>>>> It's worth calling out that recent performance
> improvements
> > to
> > > > the
> > > > > > >>>> metrics
> > > > > > >>>>>> framework mean that we no longer fetch the system time at
> > the
> > > > > > >> operator
> > > > > > >>>>>> level, but only once per task. In other words the system
> > time
> > > > is not
> > > > > > >>>> updated
> > > > > > >>>>>> between each process as a record flows through the
> > > subtopology,
> > > > so
> > > > > > >>>>>> debugging the processor-level latency via the stateleness
> > will
> > > > not
> > > > > > >> be
> > > > > > >>>>>> possible.Note that this doesn't mean the operator-level
> > > metrics
> > > > are
> > > > > > >>> not
> > > > > > >>>>>> *useful* relative to the task-level metric. Upstream
> caching
> > > > and/or
> > > > > > >>>>>> suppression
> > > > > > >>>>>> can still cause a record's staleness at some downstream
> > > stateful
> > > > > > >>>> operator
> > > > > > >>>>>> to deviate from the task-level staleness (recorded at the
> > > source
> > > > > > >>> node).
> > > > > > >>>>>>
> > > > > > >>>>>> Please let me know if you have any concerns about this
> > change.
> > > > The
> > > > > > >>>>>> KIP has been updated with the new proposal
> > > > > > >>>>>>
> > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > > bruno@confluent.io>
> > > > > > >>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi Sophie,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thank you for the KIP.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The KIP looks good to me.
> > > > > > >>>>>>>
> > > > > > >>>>>>> 50th percentile:
> > > > > > >>>>>>> I think we do not need it now. If we need it, we can add
> > it.
> > > > Here
> > > > > > >> the
> > > > > > >>>>>>> old truism applies: Adding is always easier than
> removing.
> > > > > > >>>>>>>
> > > > > > >>>>>>> processor-node-level metrics:
> > > > > > >>>>>>> I think it is good to have the staleness metrics also on
> > > > > > >>>>>>> processor-node-level. If we do not want to record them on
> > all
> > > > > > >>>>>>> processor nodes, you could restrict the recording to
> > stateful
> > > > > > >>>>>>> processor-nodes, since those are the ones that would
> > benefit
> > > > most
> > > > > > >>> from
> > > > > > >>>>>>> the staleness metrics.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best,
> > > > > > >>>>>>> Bruno
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > > > >>>> sophie@confluent.io>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Yeah, the specific reason was just to align with the
> > current
> > > > > > >>> metrics.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Is it better to conform than to be right? History has a
> > lot
> > > > to say
> > > > > > >>> on
> > > > > > >>>>>>> that
> > > > > > >>>>>>>> matter
> > > > > > >>>>>>>> but I'm not sure how much of it applies to the fine
> > details
> > > of
> > > > > > >>> metrics
> > > > > > >>>>>>>> naming :P
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> More seriously, I figured if people are looking at this
> > > metric
> > > > > > >>> they're
> > > > > > >>>>>>>> likely to
> > > > > > >>>>>>>> be looking at all the others. Then naming this one
> "-mean"
> > > > would
> > > > > > >>>> probably
> > > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the
> other
> > > > metrics
> > > > > > >>> has
> > > > > > >>>> a
> > > > > > >>>>>>>> different meaning.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> As for the percentiles, I actually like p99 (and p75)
> > > better.
> > > > I'll
> > > > > > >>>> swap
> > > > > > >>>>>>>> that out
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > > vvcephei@apache.org
> > > > > > >>>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Thanks Sophie,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a reason
> to
> > > > choose
> > > > > > >>>> "avg"
> > > > > > >>>>>>>>> instead
> > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > > > > > >> oversensitive
> > > > > > >>>>>>> because
> > > > > > >>>>>>>>> of the mistake I just made earlier, but it strikes me
> > that
> > > > "avg"
> > > > > > >> is
> > > > > > >>>>>>>>> actually
> > > > > > >>>>>>>>> ambiguous, as it refers to a family of statistics,
> > whereas
> > > > "mean"
> > > > > > >>> is
> > > > > > >>>>>>>>> specific.
> > > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with
> > "mean";
> > > > was
> > > > > > >>> that
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Regarding the names of the percentile, I actually
> > couldn't
> > > > find
> > > > > > >>> _any_
> > > > > > >>>>>>> other
> > > > > > >>>>>>>>> metrics that use percentile. Was there a reason to
> choose
> > > > "99th"
> > > > > > >> as
> > > > > > >>>>>>> opposed
> > > > > > >>>>>>>>> to "p99" or any other scheme? This is not a criticism,
> > I'm
> > > > just
> > > > > > >>>>>>> primarily
> > > > > > >>>>>>>>> asking
> > > > > > >>>>>>>>> for consistency's sake.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks again,
> > > > > > >>>>>>>>> -John
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman
> > wrote:
> > > > > > >>>>>>>>>> Alright, I can get behind adding the min metric for
> the
> > > > sake of
> > > > > > >>>>>>> pretty
> > > > > > >>>>>>>>>> graphs
> > > > > > >>>>>>>>>> (and trivial computation).
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > > > percentile)
> > > > > > >>> but I
> > > > > > >>>>>>> can
> > > > > > >>>>>>>>> see
> > > > > > >>>>>>>>>> how users might expect it and find it a bit
> disorienting
> > > > not to
> > > > > > >>>>>>> have. So
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>> updated proposed metrics are
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > > >> vvcephei@apache.org
> > > > > > >>>>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass
> > myself.
> > > I
> > > > > > >> guess
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> mean
> > > > > > >>>>>>>>>>> seems more interesting to me than the median, but
> > neither
> > > > are
> > > > > > >> as
> > > > > > >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it does
> > > round
> > > > out
> > > > > > >>> the
> > > > > > >>>>>>>>> mental
> > > > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed min
> > > along
> > > > with
> > > > > > >>> the
> > > > > > >>>>>>>>> other
> > > > > > >>>>>>>>>>> metrics to help me understand how fast the system can
> > be,
> > > > which
> > > > > > >>>>>>> helps
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> > > inexpensive
> > > > > > >> metric
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman
> > > wrote:
> > > > > > >>>>>>>>>>>> G1:
> > > > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency *up*
> > to
> > > > the
> > > > > > >>>>>>> specific
> > > > > > >>>>>>>>>>> task"
> > > > > > >>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the
> > > point
> > > > > > >> home
> > > > > > >>>>>>>>> better.
> > > > > > >>>>>>>>>>> So
> > > > > > >>>>>>>>>>>> it's the
> > > > > > >>>>>>>>>>>> "staleness of the record when it is received by that
> > > > task" --
> > > > > > >>>>>>> will
> > > > > > >>>>>>>>> update
> > > > > > >>>>>>>>>>>> the KIP
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> B1/J:
> > > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min would
> > > > actually
> > > > > > >> be
> > > > > > >>>>>>>>> useful,
> > > > > > >>>>>>>>>>>> rather than
> > > > > > >>>>>>>>>>>> just intellectually interesting. I don't feel
> strongly
> > > > that we
> > > > > > >>>>>>>>> shouldn't
> > > > > > >>>>>>>>>>>> add it, but that's
> > > > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> > > enlighten
> > > > me
> > > > > > >>>>>>> with an
> > > > > > >>>>>>>>>>>> example?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead of
> > > adding
> > > > > > >>>>>>> multiple
> > > > > > >>>>>>>>>>>> percentile
> > > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind
> of
> > > > > > >>> performance
> > > > > > >>>>>>>>> hit we
> > > > > > >>>>>>>>>>>> take on
> > > > > > >>>>>>>>>>>> metrics computation?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile
> > > > would be
> > > > > > >>> the
> > > > > > >>>>>>>>> median
> > > > > > >>>>>>>>>>>> not the
> > > > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the
> 50th
> > > > > > >>>>>>> percentile, or
> > > > > > >>>>>>>>>>> just
> > > > > > >>>>>>>>>>>> one
> > > > > > >>>>>>>>>>>> of the two?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks all!
> > > > > > >>>>>>>>>>>> Sophie
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > > > >>>>>>> vvcephei@apache.org>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> B2/G1:
> > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we report
> > > > "spans"
> > > > > > >> or
> > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the
> > > spans
> > > > are
> > > > > > >>>>>>> pretty
> > > > > > >>>>>>>>>>>>> difficult to really measure (like time spent
> waiting
> > in
> > > > the
> > > > > > >>>>>>>>> topics, or
> > > > > > >>>>>>>>>>>>> time from the event happening to the ETL producer
> > > > choosing to
> > > > > > >>>>>>> send
> > > > > > >>>>>>>>> it,
> > > > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> In other words, it's practically easier to compute
> > > spans
> > > > by
> > > > > > >>>>>>>>> subtracting
> > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies
> by
> > > > adding
> > > > > > >>>>>>> spans.
> > > > > > >>>>>>>>> You
> > > > > > >>>>>>>>>>>>> can even just consider that the span computation
> from
> > > e2e
> > > > > > >>>>>>> always
> > > > > > >>>>>>>>> just
> > > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas computing
> > e2e
> > > > > > >> latency
> > > > > > >>>>>>>>> from
> > > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to
> > the
> > > > end
> > > > > > >> you
> > > > > > >>>>>>>>> care
> > > > > > >>>>>>>>>>> about.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> It seems like people really prefer to have spans
> when
> > > > they
> > > > > > >> are
> > > > > > >>>>>>>>>>> debugging
> > > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more
> > general
> > > > > > >>>>>>> measurement
> > > > > > >>>>>>>>>>>>> that basically every person/application cares about
> > and
> > > > > > >> should
> > > > > > >>>>>>> be
> > > > > > >>>>>>>>>>>>> monitoring.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more value to
> > > more
> > > > > > >>>>>>> people if
> > > > > > >>>>>>>>> we
> > > > > > >>>>>>>>>>>>> report
> > > > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a
> > name,
> > > I
> > > > > > >> think
> > > > > > >>>>>>> I
> > > > > > >>>>>>>>> have
> > > > > > >>>>>>>>>>> no
> > > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> G2:
> > > > > > >>>>>>>>>>>>> I think the processor-node metric is nice, since
> the
> > > > inside
> > > > > > >> of
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>> task
> > > > > > >>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>> introduce a significant amount of latency in some
> > > cases.
> > > > > > >> Plus,
> > > > > > >>>>>>>>> it's a
> > > > > > >>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to know
> (for
> > > the
> > > > > > >>>>>>> purposes
> > > > > > >>>>>>>>> of
> > > > > > >>>>>>>>>>> IQ
> > > > > > >>>>>>>>>>>>> or something) how long it takes source events to
> > "show
> > > > up" at
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> store.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I think actually recording it at every processor
> > could
> > > be
> > > > > > >>>>>>>>> expensive,
> > > > > > >>>>>>>>>>> but we
> > > > > > >>>>>>>>>>>>> already record a bunch of metrics at the node
> level.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> B1:
> > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also.
> Even
> > if
> > > > it's
> > > > > > >> a
> > > > > > >>>>>>> poor
> > > > > > >>>>>>>>>>> metric
> > > > > > >>>>>>>>>>>>> for operational purposes, a lot of people might
> > expect
> > > > to see
> > > > > > >>>>>>>>> "mean".
> > > > > > >>>>>>>>>>>>> Actually,
> > > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason
> > to
> > > > leave
> > > > > > >> it
> > > > > > >>>>>>>>> off?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I might suggest:
> > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> B3:
> > > > > > >>>>>>>>>>>>> I agree we should include late records (though not
> > the
> > > > ones
> > > > > > >> we
> > > > > > >>>>>>>>> drop).
> > > > > > >>>>>>>>>>>>> It may be spiky, but only when there are
> legitimately
> > > > some
> > > > > > >>>>>>> records
> > > > > > >>>>>>>>>>> with a
> > > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole point
> of
> > > > these
> > > > > > >>>>>>> metrics.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> That's it! I don't think I have any other feedback,
> > > other
> > > > > > >> than
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>>>> request to
> > > > > > >>>>>>>>>>>>> also report "min".
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>> -John
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang
> wrote:
> > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the
> > processing
> > > > > > >>>>>>> latency
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>> task
> > > > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate
> > topics.
> > > I
> > > > have
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>>>> similar
> > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency
> > > > metric of
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>> task A
> > > > > > >>>>>>>>>>>>>> actually measures the latency of the sub-topology
> > > up-to
> > > > but
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>>>> including
> > > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency"
> > > > itself,
> > > > > > >>>>>>> since
> > > > > > >>>>>>>>>>> today we
> > > > > > >>>>>>>>>>>>>> already have several "latency" metrics already
> which
> > > are
> > > > > > >>>>>>>>> measuring
> > > > > > >>>>>>>>>>>>> elapsed
> > > > > > >>>>>>>>>>>>>> system-time for processing a record / etc, while
> > here
> > > > we are
> > > > > > >>>>>>>>>>> comparing
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > > "record-staleness"
> > > > > > >>>>>>> (note we
> > > > > > >>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case
> > > > recording at
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>>>> system-time before we start processing the record
> > > sounds
> > > > > > >> more
> > > > > > >>>>>>>>>>> natural.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > > >>>>>>> processor-node-level
> > > > > > >>>>>>>>> DEBUG
> > > > > > >>>>>>>>>>>>>> metric is worth to add, given that we already
> have a
> > > > > > >>>>>>> task-level
> > > > > > >>>>>>>>>>>>> processing
> > > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e
> > > > latency is
> > > > > > >>>>>>>>> similar
> > > > > > >>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> > > latency.
> > > > > > >>>>>>>>> Personally I
> > > > > > >>>>>>>>>>>>> think
> > > > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > > > sufficient.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Guozhang
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> > Blee-Goldman <
> > > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful
> > > gauge
> > > > for
> > > > > > >>>>>>> this
> > > > > > >>>>>>>>>>>>> specific
> > > > > > >>>>>>>>>>>>>>> metric, as
> > > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper
> > > > *bound on
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> latency
> > > > > > >>>>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles
> that
> > > > would
> > > > > > >>>>>>>>> hopefully
> > > > > > >>>>>>>>>>>>> give a
> > > > > > >>>>>>>>>>>>>>> good
> > > > > > >>>>>>>>>>>>>>> sense of what *most* records will experience, and
> > > what
> > > > > > >>>>>>> *close
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>> all*
> > > > > > >>>>>>>>>>>>>>> records
> > > > > > >>>>>>>>>>>>>>> will.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> However I'm not married to these specific numbers
> > and
> > > > > > >>>>>>> could be
> > > > > > >>>>>>>>>>>>> convinced.
> > > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing from
> > users
> > > on
> > > > > > >>>>>>> this.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop
> > > > latency" in
> > > > > > >>>>>>>>> this KIP
> > > > > > >>>>>>>>>>>>> since
> > > > > > >>>>>>>>>>>>>>> users
> > > > > > >>>>>>>>>>>>>>> can always compute it themselves by subtracting
> the
> > > > > > >>>>>>> previous
> > > > > > >>>>>>>>> node's
> > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either
> > way
> > > > since
> > > > > > >>>>>>>>> you can
> > > > > > >>>>>>>>>>>>> always
> > > > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> > > end-to-end
> > > > > > >>>>>>> latency
> > > > > > >>>>>>>>>>> feels
> > > > > > >>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> > > > > > >>>>>>> bottlenecks
> > > > > > >>>>>>>>> in the
> > > > > > >>>>>>>>>>>>>>> topology but
> > > > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes
> arecord
> > to
> > > > be
> > > > > > >>>>>>>>> reflected
> > > > > > >>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>> certain parts
> > > > > > >>>>>>>>>>>>>>> of the topology. For example this might be useful
> > for
> > > > users
> > > > > > >>>>>>>>> who are
> > > > > > >>>>>>>>>>>>>>> wondering
> > > > > > >>>>>>>>>>>>>>> roughly when a record that was just produced will
> > be
> > > > > > >>>>>>> included
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>>> their
> > > > > > >>>>>>>>>>>>> IQ
> > > > > > >>>>>>>>>>>>>>> results.
> > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but
> maybe I
> > > > didn't
> > > > > > >>>>>>> make
> > > > > > >>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>> clear
> > > > > > >>>>>>>>>>>>>>> enough
> > > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in the
> KIP.
> > > The
> > > > > > >>>>>>> short
> > > > > > >>>>>>>>>>> answer is
> > > > > > >>>>>>>>>>>>>>> "yes",
> > > > > > >>>>>>>>>>>>>>> we will include late records. I added a paragraph
> > to
> > > > the
> > > > > > >>>>>>> end
> > > > > > >>>>>>>>> of the
> > > > > > >>>>>>>>>>>>>>> Proposed
> > > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning here,
> > please
> > > > let
> > > > > > >>>>>>> me
> > > > > > >>>>>>>>> know
> > > > > > >>>>>>>>>>> if
> > > > > > >>>>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>>> have
> > > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing
> metric
> > > > > > >>>>>>>>>>> "process-latency",
> > > > > > >>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>> metric
> > > > > > >>>>>>>>>>>>>>> reflects the time for the literal Node#process
> > method
> > > > to
> > > > > > >>>>>>> run
> > > > > > >>>>>>>>>>> whereas
> > > > > > >>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>> metric
> > > > > > >>>>>>>>>>>>>>> would always be measured relative to the event
> > > > timestamp.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> That said, the naming collision there is pretty
> > > > confusing
> > > > > > >>>>>>> so
> > > > > > >>>>>>>>> I've
> > > > > > >>>>>>>>>>>>> renamed
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency"
> which I
> > > > feel
> > > > > > >>>>>>> better
> > > > > > >>>>>>>>>>>>> reflects
> > > > > > >>>>>>>>>>>>>>> the nature
> > > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E
> latency
> > > is
> > > > > > >>>>>>>>> important
> > > > > > >>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75,
> > 99
> > > > and
> > > > > > >>>>>>> max
> > > > > > >>>>>>>>>>>>> percentiles?
> > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> > percentile
> > > as
> > > > > > >>>>>>> well
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>>>>> production
> > > > > > >>>>>>>>>>>>>>>> systems.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> > cumulative,
> > > > I.E
> > > > > > >>>>>>> if a
> > > > > > >>>>>>>>>>> record
> > > > > > >>>>>>>>>>>>> goes
> > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B)
> =
> > > > > > >>>>>>> T(B->C) +
> > > > > > >>>>>>>>>>> T(A->B) +
> > > > > > >>>>>>>>>>>>>>> T(A)
> > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> > > latency,
> > > > > > >>>>>>> and T()
> > > > > > >>>>>>>>>>>>> represents
> > > > > > >>>>>>>>>>>>>>>> the time for transiting the records between two
> > > nodes,
> > > > > > >>>>>>>>> including
> > > > > > >>>>>>>>>>>>>>> processing
> > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having
> T(B->C)
> > > and
> > > > > > >>>>>>>>> T(A->B)
> > > > > > >>>>>>>>>>> are
> > > > > > >>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency",
> otherwise
> > > if
> > > > > > >>>>>>> there
> > > > > > >>>>>>>>> is a
> > > > > > >>>>>>>>>>>>> spike in
> > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the
> > same
> > > > > > >>>>>>> time.
> > > > > > >>>>>>>>> In the
> > > > > > >>>>>>>>>>>>> same
> > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when
> > the
> > > > > > >>>>>>> record
> > > > > > >>>>>>>>> exits
> > > > > > >>>>>>>>>>>>> from the
> > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this record
> > spent
> > > > > > >>>>>>> inside
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>>> funnel. Do
> > > > > > >>>>>>>>>>>>>>>> you think we could have separate treatment for
> > sink
> > > > > > >>>>>>> nodes and
> > > > > > >>>>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time
> > > > receiving
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> record
> > > > > > >>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here,
> just
> > > > want to
> > > > > > >>>>>>>>> discuss
> > > > > > >>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival
> records
> > > as
> > > > > > >>>>>>> well,
> > > > > > >>>>>>>>> they
> > > > > > >>>>>>>>>>>>> would
> > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > > out-of-order
> > > > > > >>>>>>>>> records are
> > > > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we
> also
> > > > supply
> > > > > > >>>>>>> a
> > > > > > >>>>>>>>> smooth
> > > > > > >>>>>>>>>>>>> version
> > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just take
> > > care
> > > > of
> > > > > > >>>>>>> it
> > > > > > >>>>>>>>> by
> > > > > > >>>>>>>>>>>>> themself?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't
> > discussed
> > > > its
> > > > > > >>>>>>>>> relation
> > > > > > >>>>>>>>>>>>> with our
> > > > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could you
> add
> > > > some
> > > > > > >>>>>>>>> context
> > > > > > >>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which`
> > tutorial
> > > > for
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>> best?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> > Blee-Goldman
> > > <
> > > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613
> which
> > > aims
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>> add
> > > > > > >>>>>>>>>>>>> end-to-end
> > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> --
> > > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Hey Guozhang,

Thanks for the response. I meant to ask this earlier and forgot, but do you
have any data or benchmarking results on hand for the performance hit we
took due to the metrics? Or maybe better yet, on the performance we gained
back by dropping some metrics? I think it would help us to better understand
the tradeoffs with some concrete numbers to compare.

I guess I'm inclined to want to include these metrics at the operator level
on
DEBUG since, in my mind, that's exactly what DEBUG is for: performance-
focused users will presumably run at INFO level and even those who do
use DEBUG may only do so temporarily, when they actually have something
to debug.

Of course, I'm not running a Streams app myself so all this is just
conjecture.
But it seems reasonable to interpret the two metrics levels as "minimal
useful
information" and "all information that provides insight into the system"

If we can agree there, the question becomes "does this provide any
additional
insight?" I would still argue that it does. To give a concrete example,
let's say
the user has a basic

SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK

subtopology, the application performing IQ on the transformer's state. The
user
wants to know how long it's taking records to be reflected in the results
of a query.
With only task-level metrics, they can only guess based on the e2e latency
of the
previous task and of this one. But neither of those is a good
representation of the
time it will take a record to be processed by the transformer: the previous
task's
latency will not include the time for the record to go through the
repartitioning,
and the current task's latency is measured after a suppression. The
suppression
itself will introduce significant latency and doesn't provide an accurate
estimate.
Caching would also do what it does best, messing with our understanding of
time :)

It also means you can't measure the consumption latency, so there would be
no way to tell if the e2e latency is so high because the very first
subtopology is
extremely slow, or the record was severely delayed in reaching the input
topic.
I guess my earlier proposal to include both source and sink metrics could
address this issue, but it seems to solve the problem more holistically --
and for
more complex subtopologies -- by including metrics at the processor level

On Fri, May 15, 2020 at 5:44 PM Guozhang Wang <wa...@gmail.com> wrote:

> Hey folks,
>
> I want to make a final fight on processor-node-level metrics :) Post
> KIP-444 we've actually removed a lot of node-level metrics since it is too
> expensive to record at that level and their values are not proven worth it.
> Again I'd use the "we have to either enable all or none of DEBUG metrics"
> card, e.g. if we want to look into per-task process-latency which is DEBUG,
> we'd have to record all process-node / state-store / cache level metrics.
> On the other hand, within a task we usually do not have a lot of stateful
> operator nodes and if we want to find if certain stateful nodes are the
> bottleneck of process latency, that is usually discoverable from the
> state-store level read / write latencies already. So I'd imagine that
> suppose there's a sub-topology with multiple state stores in it, if we
> found the task-level process latency is high, we can still tell which state
> stores within it is the main bottleneck from the state store read / write
> latency (I'm assuming the IO latency is always dominant compared with
> others).
>
> Guozhang
>
>
>
>
> On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
>
> > > @Matthias
> > > And that's what he says about the 95th percentile! Imagine what he
> would
> > say about the 50th :P
> >
> > I should have kept watching another 10 minutes. He gets around to
> covering
> > the 50th, and let's
> > just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
> >
> > On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org>
> wrote:
> >
> > > Thanks, Sophie!
> > >
> > > I think this makes perfect sense. It will be much more intuitive to use
> > > the metric for the stated motivation this way.  I’d be in favor of the
> > > proposal after this update.
> > >
> > > Thanks again for taking this on,
> > > -John
> > >
> > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > > Hi all,
> > > >
> > > > I'd like to clarify/modify one aspect of this KIP, which is to record
> > the
> > > > staleness
> > > > at the *completion* of the record's processing by the operator/task
> in
> > > > question,
> > > > rather than on the intake. The task-level metrics will be recorded at
> > the
> > > > sink
> > > > node instead of at the source, and the operator-level metrics will be
> > > > recorded
> > > > at the end of the operation.
> > > >
> > > > The stated purpose and intended usefulness of this KIP is to give
> > users a
> > > > way
> > > > to gauge roughly how long it takes for a record to be reflected in
> the
> > > > "results",
> > > > whether these results are being read from an output topic or through
> > IQ.
> > > To
> > > > take the IQ example, the results of a record are obviously not
> visible
> > > until
> > > > *after* that node has finished processing it. The staleness, while
> > still
> > > > potentially
> > > > useful as it can impact the *way* a record is processed in a stateful
> > and
> > > > time-
> > > > dependent operator, is not part of the problem this KIP specifically
> > set
> > > out
> > > > to solve.
> > > >
> > > > In light of this I think it's appropriate to revert the name change
> > back
> > > to
> > > > include
> > > > latency, since "staleness" as described above only makes sense when
> > > > measuring
> > > > relative to the arrival of a record at a task/node. I'd propose to
> save
> > > the
> > > > term
> > > > "staleness" for that particular meaning, and adopt Matthias's
> > suggestion
> > > of
> > > > "record-e2e-latency" for this.
> > > >
> > > > Thanks for hanging in there all. Please let me know if you have any
> > > > concerns about
> > > > this change!
> > > >
> > > > Sophie
> > > >
> > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org>
> > > wrote:
> > > >
> > > > > I am also happy with max/min/99/90. And I buy your naming argument
> > > about
> > > > > staleness vs latency.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > > Hey Sophie,
> > > > > >
> > > > > > 90/99/min/max make sense to me.
> > > > > >
> > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > > sophie@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > >> @Matthias
> > > > > >> Regarding tracking the 50th percentile, I'll refer you to the
> 4:53
> > > mark
> > > > > of
> > > > > >> the video
> > > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > > >>
> > > > > >> And that's what he says about the 95th percentile! Imagine what
> he
> > > would
> > > > > >> say about
> > > > > >> the 50th :P
> > > > > >>
> > > > > >> But seriously, since we can't seem to agree that the mean or
> 50th
> > > > > >> percentile is actually
> > > > > >> useful I'm inclined to resurrect my original proposal, neither.
> > But
> > > I
> > > > > think
> > > > > >> that's a good
> > > > > >> argument against the 75th, which I admittedly chose somewhat
> > > > > arbitrarily as
> > > > > >> an
> > > > > >> intermediate between the 50th and the higher percentiles. How
> > about:
> > > > > >>
> > > > > >> -max
> > > > > >> -p99
> > > > > >> -p90
> > > > > >> -min
> > > > > >>
> > > > > >> with p50/mean still up for debate if anyone feels strongly for
> > > either of
> > > > > >> them.
> > > > > >>
> > > > > >> Regarding the name, I've already flip-flopped on this so I'm
> > > definitely
> > > > > >> still open to
> > > > > >> further arguments. But the reason for changing it from
> > > > > end-to-end-latency
> > > > > >> (which
> > > > > >> is similar to what you propose) is that this metric technically
> > > reflects
> > > > > >> how old (ie how "stale")
> > > > > >> the record is when it's *received* by the operator, not when
> it's
> > > > > processed
> > > > > >> by the operator.
> > > > > >> It seemed like there was the potential for confusion that
> > > > > >> "end-to-end-latency" might
> > > > > >> represent the latency from the event creation to the time the
> > > processor
> > > > > is
> > > > > >> done
> > > > > >> processing it.
> > > > > >>
> > > > > >> @John
> > > > > >> I'd rather err on the side of "not-enough" metrics as we can
> > always
> > > add
> > > > > >> this to the
> > > > > >> stateless metrics later on. If we decide to measure the time at
> > > every
> > > > > node
> > > > > >> and don't
> > > > > >> find any evidence of a serious performance impact, and users
> > > indicate
> > > > > they
> > > > > >> would
> > > > > >> like to see this metric at all nodes, then we can easily start
> > > reporting
> > > > > >> them as well.
> > > > > >> WDYT?
> > > > > >>
> > > > > >> That said, sink nodes seem like a reasonable exception to the
> > rule.
> > > > > >> Obviously users
> > > > > >> should be able to detect the time when the record reaches the
> > output
> > > > > topic
> > > > > >> but that
> > > > > >> still leaves a gap in understanding how long the production
> > latency
> > > was.
> > > > > >> This mirrors
> > > > > >> the consumption latency that is exposed by the task-level
> metrics,
> > > which
> > > > > >> are measured
> > > > > >> at the source node. For good symmetry what if we actually expose
> > > both
> > > > > the
> > > > > >> source
> > > > > >> and sink latency at the task-level? ie report both sets of
> > > statistical
> > > > > >> measurements with
> > > > > >> the additional tag -source/-sink
> > > > > >>
> > > > > >> @Bill
> > > > > >> Thanks for the comment about regarding the min! I hadn't
> > considered
> > > that
> > > > > >> and it's
> > > > > >> quite useful to think about how and what is useful from a users
> > > point of
> > > > > >> view.
> > > > > >>
> > > > > >> Regarding your second. point, I'm inclined to leave that as an
> > > > > >> implementation detail
> > > > > >> but my take would be that the user should be allowed to control
> > the
> > > > > record
> > > > > >> timestamp
> > > > > >> used for this with the timestamp extractor. My impression is
> that
> > > users
> > > > > may
> > > > > >> often embed
> > > > > >> the actual event time in the payload for whatever reason, and
> this
> > > > > >> represents the "true"
> > > > > >> timestamp as far as the Streams topology is concerned.
> > > > > >>
> > > > > >>
> > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bbejeck@gmail.com
> >
> > > wrote:
> > > > > >>
> > > > > >>> Thanks for the KIP, Sophie, this will be a useful metric to
> add.
> > > > > >>>
> > > > > >>> Regarding tracking min, I  think it could be valuable for users
> > to
> > > > > >> discern
> > > > > >>> which part of their topologies are more efficient since this
> is a
> > > > > >>> task-level metric.  I realize everyone seems to be on board
> with
> > > > > >> including
> > > > > >>> min anyway, but I wanted to add my 2 cents on this topic should
> > we
> > > > > decide
> > > > > >>> to revisit adding min or not.
> > > > > >>>
> > > > > >>> I do have a question regarding the calculation of staleness.
> > > > > >>> Is there going to be a consideration for timestamp extractors?
> > > Users
> > > > > >> could
> > > > > >>> prefer to use a timestamp embedded in the payload, and it could
> > > skew
> > > > > the
> > > > > >>> measurements.
> > > > > >>> I was wondering if we should specify in the KIP if setting the
> > > arrival
> > > > > >> time
> > > > > >>> is always going to come from the record timestamp, or is this
> an
> > > > > >>> implementation detail we can cover in the PR?
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>> Bill
> > > > > >>>
> > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> > mjsax@apache.org>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>> Thanks for the KIP Sophie.
> > > > > >>>>
> > > > > >>>> I think it's not useful to record the avg/mean; it sensitive
> to
> > > > > >>>> outliers. We should rather track the median (50th percentile).
> > > > > >>>>
> > > > > >>>> Not sure if tracking min is useful, but I am also ok to track
> > it.
> > > > > >>>>
> > > > > >>>> However, I find it odd to track 75th percentile. Standard
> > measures
> > > > > >> would
> > > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe
> picking
> > > 90th
> > > > > >>>> might be more useful?
> > > > > >>>>
> > > > > >>>> About the name: "staleness" wound really odd, and if fact the
> > > metric
> > > > > >>>> does capture "latency" so we should call it "latency". I
> > > understand
> > > > > the
> > > > > >>>> issue that we already have a latency metric. So maybe we could
> > > call it
> > > > > >>>> `record-e2e-latency-*` ?
> > > > > >>>>
> > > > > >>>> While I agree that we should include out-or-order data (the
> KIP
> > > should
> > > > > >>>> talk about `out-of-order` data, not `late` data; data is only
> > > `late`
> > > > > if
> > > > > >>>> it's out-of-order and if it's dropped), I don't really
> > understand
> > > why
> > > > > >>>> the new metric would help to configure grace period or
> retention
> > > time?
> > > > > >>>> As you mention in the KIP, both are define as max difference
> of
> > > > > >>>> `event-time - stream-time` and thus the new metric that takes
> > > > > >>>> system-/wallclock-time into account does not seem to help at
> > all.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Btw: there is a great talk about "How NOT to Measure Latency"
> by
> > > Gil
> > > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> -Matthias
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > > >>>>> Hi Sophie,
> > > > > >>>>>
> > > > > >>>>> It seems like there would still be plenty of use cases for
> > > recording
> > > > > >>>>> this metric at all processors and not just stateful ones, but
> > I'm
> > > > > >> happy
> > > > > >>>>> to suspend my arguments for now. Since you're proposing to
> keep
> > > > > >>>>> them at the processor-node level, it will be seamless later
> to
> > > add
> > > > > >>>>> in the stateless processors if we want. As a wise man once
> > said,
> > > > > >>>>> "Adding is always easier than removing."
> > > > > >>>>>
> > > > > >>>>> Regarding the time measurement, it's an implementation detail
> > > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd
> greatly
> > > > > >>>>> prefer to measure the system time again when recording the
> > > > > >>>>> metric. I don't think we've seen any evidence that proves
> this
> > > > > >>>>> would harm performance, and the amount of inaccuracy using
> > > > > >>>>> the cached system time could incur is actually substantial.
> > But,
> > > > > >>>>> if you want to just "not mention this" in the KIP, we can
> defer
> > > to
> > > > > >>>>> the actual PR discussion, at which time we're in a better
> > > position
> > > > > >>>>> to use benchmarks, etc., to make the call.
> > > > > >>>>>
> > > > > >>>>> Along the lines of the measurement accuracy discussion, one
> > > > > >>>>> minor thought I had is that maybe we should consider
> measuring
> > > > > >>>>> the task staleness metric at the sink, rather than the
> source,
> > so
> > > > > >> that
> > > > > >>>>> it includes the processing latency of the task itself, not
> just
> > > the
> > > > > >>>> latency
> > > > > >>>>> of everything up to, but not including, the task (which seems
> > > > > >> confusing
> > > > > >>>>> for users). I guess this could also be an implementation
> > detail,
> > > > > >>> though.
> > > > > >>>>>
> > > > > >>>>> Thanks for the update,
> > > > > >>>>> -John
> > > > > >>>>>
> > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > > > > >>>>>> Hey all,
> > > > > >>>>>>
> > > > > >>>>>> After discussing with Bruno I'd like to propose a small
> > > amendment,
> > > > > >>>>>> which is to record the processor-node-level metrics only for
> > > > > >>> *stateful*
> > > > > >>>>>> *operators*. They would still be considered a
> > > "processor-node-level"
> > > > > >>>>>> metric and not a "state-store-level" metric as the staleness
> > is
> > > > > >> still
> > > > > >>>>>> a property of the node rather than of the state itself.
> > > However, it
> > > > > >>>> seems
> > > > > >>>>>> that this information is primarily useful for stateful
> > operators
> > > > > >> that
> > > > > >>>> might
> > > > > >>>>>> be exposing state via IQ or otherwise dependent on the
> record
> > > time
> > > > > >>>>>> unlike a stateless operator.
> > > > > >>>>>>
> > > > > >>>>>> It's worth calling out that recent performance improvements
> to
> > > the
> > > > > >>>> metrics
> > > > > >>>>>> framework mean that we no longer fetch the system time at
> the
> > > > > >> operator
> > > > > >>>>>> level, but only once per task. In other words the system
> time
> > > is not
> > > > > >>>> updated
> > > > > >>>>>> between each process as a record flows through the
> > subtopology,
> > > so
> > > > > >>>>>> debugging the processor-level latency via the stateleness
> will
> > > not
> > > > > >> be
> > > > > >>>>>> possible.Note that this doesn't mean the operator-level
> > metrics
> > > are
> > > > > >>> not
> > > > > >>>>>> *useful* relative to the task-level metric. Upstream caching
> > > and/or
> > > > > >>>>>> suppression
> > > > > >>>>>> can still cause a record's staleness at some downstream
> > stateful
> > > > > >>>> operator
> > > > > >>>>>> to deviate from the task-level staleness (recorded at the
> > source
> > > > > >>> node).
> > > > > >>>>>>
> > > > > >>>>>> Please let me know if you have any concerns about this
> change.
> > > The
> > > > > >>>>>> KIP has been updated with the new proposal
> > > > > >>>>>>
> > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > > bruno@confluent.io>
> > > > > >>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi Sophie,
> > > > > >>>>>>>
> > > > > >>>>>>> Thank you for the KIP.
> > > > > >>>>>>>
> > > > > >>>>>>> The KIP looks good to me.
> > > > > >>>>>>>
> > > > > >>>>>>> 50th percentile:
> > > > > >>>>>>> I think we do not need it now. If we need it, we can add
> it.
> > > Here
> > > > > >> the
> > > > > >>>>>>> old truism applies: Adding is always easier than removing.
> > > > > >>>>>>>
> > > > > >>>>>>> processor-node-level metrics:
> > > > > >>>>>>> I think it is good to have the staleness metrics also on
> > > > > >>>>>>> processor-node-level. If we do not want to record them on
> all
> > > > > >>>>>>> processor nodes, you could restrict the recording to
> stateful
> > > > > >>>>>>> processor-nodes, since those are the ones that would
> benefit
> > > most
> > > > > >>> from
> > > > > >>>>>>> the staleness metrics.
> > > > > >>>>>>>
> > > > > >>>>>>> Best,
> > > > > >>>>>>> Bruno
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > > >>>> sophie@confluent.io>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yeah, the specific reason was just to align with the
> current
> > > > > >>> metrics.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Is it better to conform than to be right? History has a
> lot
> > > to say
> > > > > >>> on
> > > > > >>>>>>> that
> > > > > >>>>>>>> matter
> > > > > >>>>>>>> but I'm not sure how much of it applies to the fine
> details
> > of
> > > > > >>> metrics
> > > > > >>>>>>>> naming :P
> > > > > >>>>>>>>
> > > > > >>>>>>>> More seriously, I figured if people are looking at this
> > metric
> > > > > >>> they're
> > > > > >>>>>>>> likely to
> > > > > >>>>>>>> be looking at all the others. Then naming this one "-mean"
> > > would
> > > > > >>>> probably
> > > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the other
> > > metrics
> > > > > >>> has
> > > > > >>>> a
> > > > > >>>>>>>> different meaning.
> > > > > >>>>>>>>
> > > > > >>>>>>>> As for the percentiles, I actually like p99 (and p75)
> > better.
> > > I'll
> > > > > >>>> swap
> > > > > >>>>>>>> that out
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > > vvcephei@apache.org
> > > > > >>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Thanks Sophie,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a reason to
> > > choose
> > > > > >>>> "avg"
> > > > > >>>>>>>>> instead
> > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > > > > >> oversensitive
> > > > > >>>>>>> because
> > > > > >>>>>>>>> of the mistake I just made earlier, but it strikes me
> that
> > > "avg"
> > > > > >> is
> > > > > >>>>>>>>> actually
> > > > > >>>>>>>>> ambiguous, as it refers to a family of statistics,
> whereas
> > > "mean"
> > > > > >>> is
> > > > > >>>>>>>>> specific.
> > > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with
> "mean";
> > > was
> > > > > >>> that
> > > > > >>>>>>> the
> > > > > >>>>>>>>> reason? If so, I'm +1.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Regarding the names of the percentile, I actually
> couldn't
> > > find
> > > > > >>> _any_
> > > > > >>>>>>> other
> > > > > >>>>>>>>> metrics that use percentile. Was there a reason to choose
> > > "99th"
> > > > > >> as
> > > > > >>>>>>> opposed
> > > > > >>>>>>>>> to "p99" or any other scheme? This is not a criticism,
> I'm
> > > just
> > > > > >>>>>>> primarily
> > > > > >>>>>>>>> asking
> > > > > >>>>>>>>> for consistency's sake.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks again,
> > > > > >>>>>>>>> -John
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman
> wrote:
> > > > > >>>>>>>>>> Alright, I can get behind adding the min metric for the
> > > sake of
> > > > > >>>>>>> pretty
> > > > > >>>>>>>>>> graphs
> > > > > >>>>>>>>>> (and trivial computation).
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > > percentile)
> > > > > >>> but I
> > > > > >>>>>>> can
> > > > > >>>>>>>>> see
> > > > > >>>>>>>>>> how users might expect it and find it a bit disorienting
> > > not to
> > > > > >>>>>>> have. So
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>> updated proposed metrics are
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > > >> vvcephei@apache.org
> > > > > >>>>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass
> myself.
> > I
> > > > > >> guess
> > > > > >>>>>>> the
> > > > > >>>>>>>>> mean
> > > > > >>>>>>>>>>> seems more interesting to me than the median, but
> neither
> > > are
> > > > > >> as
> > > > > >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it does
> > round
> > > out
> > > > > >>> the
> > > > > >>>>>>>>> mental
> > > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed min
> > along
> > > with
> > > > > >>> the
> > > > > >>>>>>>>> other
> > > > > >>>>>>>>>>> metrics to help me understand how fast the system can
> be,
> > > which
> > > > > >>>>>>> helps
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> > inexpensive
> > > > > >> metric
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman
> > wrote:
> > > > > >>>>>>>>>>>> G1:
> > > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency *up*
> to
> > > the
> > > > > >>>>>>> specific
> > > > > >>>>>>>>>>> task"
> > > > > >>>>>>>>>>>> but
> > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the
> > point
> > > > > >> home
> > > > > >>>>>>>>> better.
> > > > > >>>>>>>>>>> So
> > > > > >>>>>>>>>>>> it's the
> > > > > >>>>>>>>>>>> "staleness of the record when it is received by that
> > > task" --
> > > > > >>>>>>> will
> > > > > >>>>>>>>> update
> > > > > >>>>>>>>>>>> the KIP
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> B1/J:
> > > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min would
> > > actually
> > > > > >> be
> > > > > >>>>>>>>> useful,
> > > > > >>>>>>>>>>>> rather than
> > > > > >>>>>>>>>>>> just intellectually interesting. I don't feel strongly
> > > that we
> > > > > >>>>>>>>> shouldn't
> > > > > >>>>>>>>>>>> add it, but that's
> > > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> > enlighten
> > > me
> > > > > >>>>>>> with an
> > > > > >>>>>>>>>>>> example?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead of
> > adding
> > > > > >>>>>>> multiple
> > > > > >>>>>>>>>>>> percentile
> > > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
> > > > > >>> performance
> > > > > >>>>>>>>> hit we
> > > > > >>>>>>>>>>>> take on
> > > > > >>>>>>>>>>>> metrics computation?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile
> > > would be
> > > > > >>> the
> > > > > >>>>>>>>> median
> > > > > >>>>>>>>>>>> not the
> > > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > > > > >>>>>>> percentile, or
> > > > > >>>>>>>>>>> just
> > > > > >>>>>>>>>>>> one
> > > > > >>>>>>>>>>>> of the two?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks all!
> > > > > >>>>>>>>>>>> Sophie
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > > >>>>>>> vvcephei@apache.org>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> B2/G1:
> > > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we report
> > > "spans"
> > > > > >> or
> > > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the
> > spans
> > > are
> > > > > >>>>>>> pretty
> > > > > >>>>>>>>>>>>> difficult to really measure (like time spent waiting
> in
> > > the
> > > > > >>>>>>>>> topics, or
> > > > > >>>>>>>>>>>>> time from the event happening to the ETL producer
> > > choosing to
> > > > > >>>>>>> send
> > > > > >>>>>>>>> it,
> > > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> In other words, it's practically easier to compute
> > spans
> > > by
> > > > > >>>>>>>>> subtracting
> > > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by
> > > adding
> > > > > >>>>>>> spans.
> > > > > >>>>>>>>> You
> > > > > >>>>>>>>>>>>> can even just consider that the span computation from
> > e2e
> > > > > >>>>>>> always
> > > > > >>>>>>>>> just
> > > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas computing
> e2e
> > > > > >> latency
> > > > > >>>>>>>>> from
> > > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to
> the
> > > end
> > > > > >> you
> > > > > >>>>>>>>> care
> > > > > >>>>>>>>>>> about.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> It seems like people really prefer to have spans when
> > > they
> > > > > >> are
> > > > > >>>>>>>>>>> debugging
> > > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more
> general
> > > > > >>>>>>> measurement
> > > > > >>>>>>>>>>>>> that basically every person/application cares about
> and
> > > > > >> should
> > > > > >>>>>>> be
> > > > > >>>>>>>>>>>>> monitoring.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Altogether, it really seem to provide more value to
> > more
> > > > > >>>>>>> people if
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>>>>> report
> > > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a
> name,
> > I
> > > > > >> think
> > > > > >>>>>>> I
> > > > > >>>>>>>>> have
> > > > > >>>>>>>>>>> no
> > > > > >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> G2:
> > > > > >>>>>>>>>>>>> I think the processor-node metric is nice, since the
> > > inside
> > > > > >> of
> > > > > >>>>>>> a
> > > > > >>>>>>>>> task
> > > > > >>>>>>>>>>> can
> > > > > >>>>>>>>>>>>> introduce a significant amount of latency in some
> > cases.
> > > > > >> Plus,
> > > > > >>>>>>>>> it's a
> > > > > >>>>>>>>>>> more
> > > > > >>>>>>>>>>>>> direct measurement, if you really wanted to know (for
> > the
> > > > > >>>>>>> purposes
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>>> IQ
> > > > > >>>>>>>>>>>>> or something) how long it takes source events to
> "show
> > > up" at
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> store.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I think actually recording it at every processor
> could
> > be
> > > > > >>>>>>>>> expensive,
> > > > > >>>>>>>>>>> but we
> > > > > >>>>>>>>>>>>> already record a bunch of metrics at the node level.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> B1:
> > > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also. Even
> if
> > > it's
> > > > > >> a
> > > > > >>>>>>> poor
> > > > > >>>>>>>>>>> metric
> > > > > >>>>>>>>>>>>> for operational purposes, a lot of people might
> expect
> > > to see
> > > > > >>>>>>>>> "mean".
> > > > > >>>>>>>>>>>>> Actually,
> > > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason
> to
> > > leave
> > > > > >> it
> > > > > >>>>>>>>> off?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I might suggest:
> > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> B3:
> > > > > >>>>>>>>>>>>> I agree we should include late records (though not
> the
> > > ones
> > > > > >> we
> > > > > >>>>>>>>> drop).
> > > > > >>>>>>>>>>>>> It may be spiky, but only when there are legitimately
> > > some
> > > > > >>>>>>> records
> > > > > >>>>>>>>>>> with a
> > > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole point of
> > > these
> > > > > >>>>>>> metrics.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> That's it! I don't think I have any other feedback,
> > other
> > > > > >> than
> > > > > >>>>>>> a
> > > > > >>>>>>>>>>> request to
> > > > > >>>>>>>>>>>>> also report "min".
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>> -John
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the
> processing
> > > > > >>>>>>> latency
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>> task
> > > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate
> topics.
> > I
> > > have
> > > > > >>>>>>> a
> > > > > >>>>>>>>>>> similar
> > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency
> > > metric of
> > > > > >>>>>>> a
> > > > > >>>>>>>>> task A
> > > > > >>>>>>>>>>>>>> actually measures the latency of the sub-topology
> > up-to
> > > but
> > > > > >>>>>>> not
> > > > > >>>>>>>>>>> including
> > > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency"
> > > itself,
> > > > > >>>>>>> since
> > > > > >>>>>>>>>>> today we
> > > > > >>>>>>>>>>>>>> already have several "latency" metrics already which
> > are
> > > > > >>>>>>>>> measuring
> > > > > >>>>>>>>>>>>> elapsed
> > > > > >>>>>>>>>>>>>> system-time for processing a record / etc, while
> here
> > > we are
> > > > > >>>>>>>>>>> comparing
> > > > > >>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> > "record-staleness"
> > > > > >>>>>>> (note we
> > > > > >>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case
> > > recording at
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>>>> system-time before we start processing the record
> > sounds
> > > > > >> more
> > > > > >>>>>>>>>>> natural.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > > >>>>>>> processor-node-level
> > > > > >>>>>>>>> DEBUG
> > > > > >>>>>>>>>>>>>> metric is worth to add, given that we already have a
> > > > > >>>>>>> task-level
> > > > > >>>>>>>>>>>>> processing
> > > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e
> > > latency is
> > > > > >>>>>>>>> similar
> > > > > >>>>>>>>>>> to
> > > > > >>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> > latency.
> > > > > >>>>>>>>> Personally I
> > > > > >>>>>>>>>>>>> think
> > > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > > sufficient.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Guozhang
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie
> Blee-Goldman <
> > > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful
> > gauge
> > > for
> > > > > >>>>>>> this
> > > > > >>>>>>>>>>>>> specific
> > > > > >>>>>>>>>>>>>>> metric, as
> > > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper
> > > *bound on
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> latency
> > > > > >>>>>>>>>>>>> you
> > > > > >>>>>>>>>>>>>>> can
> > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that
> > > would
> > > > > >>>>>>>>> hopefully
> > > > > >>>>>>>>>>>>> give a
> > > > > >>>>>>>>>>>>>>> good
> > > > > >>>>>>>>>>>>>>> sense of what *most* records will experience, and
> > what
> > > > > >>>>>>> *close
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>> all*
> > > > > >>>>>>>>>>>>>>> records
> > > > > >>>>>>>>>>>>>>> will.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> However I'm not married to these specific numbers
> and
> > > > > >>>>>>> could be
> > > > > >>>>>>>>>>>>> convinced.
> > > > > >>>>>>>>>>>>>>> Would be especially interested in hearing from
> users
> > on
> > > > > >>>>>>> this.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop
> > > latency" in
> > > > > >>>>>>>>> this KIP
> > > > > >>>>>>>>>>>>> since
> > > > > >>>>>>>>>>>>>>> users
> > > > > >>>>>>>>>>>>>>> can always compute it themselves by subtracting the
> > > > > >>>>>>> previous
> > > > > >>>>>>>>> node's
> > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either
> way
> > > since
> > > > > >>>>>>>>> you can
> > > > > >>>>>>>>>>>>> always
> > > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> > end-to-end
> > > > > >>>>>>> latency
> > > > > >>>>>>>>>>> feels
> > > > > >>>>>>>>>>>>> more
> > > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> > > > > >>>>>>> bottlenecks
> > > > > >>>>>>>>> in the
> > > > > >>>>>>>>>>>>>>> topology but
> > > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes arecord
> to
> > > be
> > > > > >>>>>>>>> reflected
> > > > > >>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>>> certain parts
> > > > > >>>>>>>>>>>>>>> of the topology. For example this might be useful
> for
> > > users
> > > > > >>>>>>>>> who are
> > > > > >>>>>>>>>>>>>>> wondering
> > > > > >>>>>>>>>>>>>>> roughly when a record that was just produced will
> be
> > > > > >>>>>>> included
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>> their
> > > > > >>>>>>>>>>>>> IQ
> > > > > >>>>>>>>>>>>>>> results.
> > > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I
> > > didn't
> > > > > >>>>>>> make
> > > > > >>>>>>>>>>> that
> > > > > >>>>>>>>>>>>> clear
> > > > > >>>>>>>>>>>>>>> enough
> > > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP.
> > The
> > > > > >>>>>>> short
> > > > > >>>>>>>>>>> answer is
> > > > > >>>>>>>>>>>>>>> "yes",
> > > > > >>>>>>>>>>>>>>> we will include late records. I added a paragraph
> to
> > > the
> > > > > >>>>>>> end
> > > > > >>>>>>>>> of the
> > > > > >>>>>>>>>>>>>>> Proposed
> > > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning here,
> please
> > > let
> > > > > >>>>>>> me
> > > > > >>>>>>>>> know
> > > > > >>>>>>>>>>> if
> > > > > >>>>>>>>>>>>> you
> > > > > >>>>>>>>>>>>>>> have
> > > > > >>>>>>>>>>>>>>> any concerns.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > > > > >>>>>>>>>>> "process-latency",
> > > > > >>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>> metric
> > > > > >>>>>>>>>>>>>>> reflects the time for the literal Node#process
> method
> > > to
> > > > > >>>>>>> run
> > > > > >>>>>>>>>>> whereas
> > > > > >>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>> metric
> > > > > >>>>>>>>>>>>>>> would always be measured relative to the event
> > > timestamp.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> That said, the naming collision there is pretty
> > > confusing
> > > > > >>>>>>> so
> > > > > >>>>>>>>> I've
> > > > > >>>>>>>>>>>>> renamed
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I
> > > feel
> > > > > >>>>>>> better
> > > > > >>>>>>>>>>>>> reflects
> > > > > >>>>>>>>>>>>>>> the nature
> > > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency
> > is
> > > > > >>>>>>>>> important
> > > > > >>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75,
> 99
> > > and
> > > > > >>>>>>> max
> > > > > >>>>>>>>>>>>> percentiles?
> > > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90
> percentile
> > as
> > > > > >>>>>>> well
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>>>> production
> > > > > >>>>>>>>>>>>>>>> systems.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is
> cumulative,
> > > I.E
> > > > > >>>>>>> if a
> > > > > >>>>>>>>>>> record
> > > > > >>>>>>>>>>>>> goes
> > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > > > > >>>>>>> T(B->C) +
> > > > > >>>>>>>>>>> T(A->B) +
> > > > > >>>>>>>>>>>>>>> T(A)
> > > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> > latency,
> > > > > >>>>>>> and T()
> > > > > >>>>>>>>>>>>> represents
> > > > > >>>>>>>>>>>>>>>> the time for transiting the records between two
> > nodes,
> > > > > >>>>>>>>> including
> > > > > >>>>>>>>>>>>>>> processing
> > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C)
> > and
> > > > > >>>>>>>>> T(A->B)
> > > > > >>>>>>>>>>> are
> > > > > >>>>>>>>>>>>> more
> > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise
> > if
> > > > > >>>>>>> there
> > > > > >>>>>>>>> is a
> > > > > >>>>>>>>>>>>> spike in
> > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the
> same
> > > > > >>>>>>> time.
> > > > > >>>>>>>>> In the
> > > > > >>>>>>>>>>>>> same
> > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when
> the
> > > > > >>>>>>> record
> > > > > >>>>>>>>> exits
> > > > > >>>>>>>>>>>>> from the
> > > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this record
> spent
> > > > > >>>>>>> inside
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>>> funnel. Do
> > > > > >>>>>>>>>>>>>>>> you think we could have separate treatment for
> sink
> > > > > >>>>>>> nodes and
> > > > > >>>>>>>>>>> other
> > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time
> > > receiving
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> record
> > > > > >>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just
> > > want to
> > > > > >>>>>>>>> discuss
> > > > > >>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records
> > as
> > > > > >>>>>>> well,
> > > > > >>>>>>>>> they
> > > > > >>>>>>>>>>>>> would
> > > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> > out-of-order
> > > > > >>>>>>>>> records are
> > > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we also
> > > supply
> > > > > >>>>>>> a
> > > > > >>>>>>>>> smooth
> > > > > >>>>>>>>>>>>> version
> > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just take
> > care
> > > of
> > > > > >>>>>>> it
> > > > > >>>>>>>>> by
> > > > > >>>>>>>>>>>>> themself?
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't
> discussed
> > > its
> > > > > >>>>>>>>> relation
> > > > > >>>>>>>>>>>>> with our
> > > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could you add
> > > some
> > > > > >>>>>>>>> context
> > > > > >>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which`
> tutorial
> > > for
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>> best?
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Boyang
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie
> Blee-Goldman
> > <
> > > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which
> > aims
> > > > > >>>>>>> to
> > > > > >>>>>>>>> add
> > > > > >>>>>>>>>>>>> end-to-end
> > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>> Sophie
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> --
> > > > > >>>>>>>>>>>>>> -- Guozhang
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
Hey folks,

I want to make a final fight on processor-node-level metrics :) Post
KIP-444 we've actually removed a lot of node-level metrics since it is too
expensive to record at that level and their values are not proven worth it.
Again I'd use the "we have to either enable all or none of DEBUG metrics"
card, e.g. if we want to look into per-task process-latency which is DEBUG,
we'd have to record all process-node / state-store / cache level metrics.
On the other hand, within a task we usually do not have a lot of stateful
operator nodes and if we want to find if certain stateful nodes are the
bottleneck of process latency, that is usually discoverable from the
state-store level read / write latencies already. So I'd imagine that
suppose there's a sub-topology with multiple state stores in it, if we
found the task-level process latency is high, we can still tell which state
stores within it is the main bottleneck from the state store read / write
latency (I'm assuming the IO latency is always dominant compared with
others).

Guozhang




On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> > @Matthias
> > And that's what he says about the 95th percentile! Imagine what he would
> say about the 50th :P
>
> I should have kept watching another 10 minutes. He gets around to covering
> the 50th, and let's
> just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786
>
> On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org> wrote:
>
> > Thanks, Sophie!
> >
> > I think this makes perfect sense. It will be much more intuitive to use
> > the metric for the stated motivation this way.  I’d be in favor of the
> > proposal after this update.
> >
> > Thanks again for taking this on,
> > -John
> >
> > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > > Hi all,
> > >
> > > I'd like to clarify/modify one aspect of this KIP, which is to record
> the
> > > staleness
> > > at the *completion* of the record's processing by the operator/task in
> > > question,
> > > rather than on the intake. The task-level metrics will be recorded at
> the
> > > sink
> > > node instead of at the source, and the operator-level metrics will be
> > > recorded
> > > at the end of the operation.
> > >
> > > The stated purpose and intended usefulness of this KIP is to give
> users a
> > > way
> > > to gauge roughly how long it takes for a record to be reflected in the
> > > "results",
> > > whether these results are being read from an output topic or through
> IQ.
> > To
> > > take the IQ example, the results of a record are obviously not visible
> > until
> > > *after* that node has finished processing it. The staleness, while
> still
> > > potentially
> > > useful as it can impact the *way* a record is processed in a stateful
> and
> > > time-
> > > dependent operator, is not part of the problem this KIP specifically
> set
> > out
> > > to solve.
> > >
> > > In light of this I think it's appropriate to revert the name change
> back
> > to
> > > include
> > > latency, since "staleness" as described above only makes sense when
> > > measuring
> > > relative to the arrival of a record at a task/node. I'd propose to save
> > the
> > > term
> > > "staleness" for that particular meaning, and adopt Matthias's
> suggestion
> > of
> > > "record-e2e-latency" for this.
> > >
> > > Thanks for hanging in there all. Please let me know if you have any
> > > concerns about
> > > this change!
> > >
> > > Sophie
> > >
> > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > > > I am also happy with max/min/99/90. And I buy your naming argument
> > about
> > > > staleness vs latency.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > > Hey Sophie,
> > > > >
> > > > > 90/99/min/max make sense to me.
> > > > >
> > > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > > sophie@confluent.io>
> > > > > wrote:
> > > > >
> > > > >> @Matthias
> > > > >> Regarding tracking the 50th percentile, I'll refer you to the 4:53
> > mark
> > > > of
> > > > >> the video
> > > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > > >>
> > > > >> And that's what he says about the 95th percentile! Imagine what he
> > would
> > > > >> say about
> > > > >> the 50th :P
> > > > >>
> > > > >> But seriously, since we can't seem to agree that the mean or 50th
> > > > >> percentile is actually
> > > > >> useful I'm inclined to resurrect my original proposal, neither.
> But
> > I
> > > > think
> > > > >> that's a good
> > > > >> argument against the 75th, which I admittedly chose somewhat
> > > > arbitrarily as
> > > > >> an
> > > > >> intermediate between the 50th and the higher percentiles. How
> about:
> > > > >>
> > > > >> -max
> > > > >> -p99
> > > > >> -p90
> > > > >> -min
> > > > >>
> > > > >> with p50/mean still up for debate if anyone feels strongly for
> > either of
> > > > >> them.
> > > > >>
> > > > >> Regarding the name, I've already flip-flopped on this so I'm
> > definitely
> > > > >> still open to
> > > > >> further arguments. But the reason for changing it from
> > > > end-to-end-latency
> > > > >> (which
> > > > >> is similar to what you propose) is that this metric technically
> > reflects
> > > > >> how old (ie how "stale")
> > > > >> the record is when it's *received* by the operator, not when it's
> > > > processed
> > > > >> by the operator.
> > > > >> It seemed like there was the potential for confusion that
> > > > >> "end-to-end-latency" might
> > > > >> represent the latency from the event creation to the time the
> > processor
> > > > is
> > > > >> done
> > > > >> processing it.
> > > > >>
> > > > >> @John
> > > > >> I'd rather err on the side of "not-enough" metrics as we can
> always
> > add
> > > > >> this to the
> > > > >> stateless metrics later on. If we decide to measure the time at
> > every
> > > > node
> > > > >> and don't
> > > > >> find any evidence of a serious performance impact, and users
> > indicate
> > > > they
> > > > >> would
> > > > >> like to see this metric at all nodes, then we can easily start
> > reporting
> > > > >> them as well.
> > > > >> WDYT?
> > > > >>
> > > > >> That said, sink nodes seem like a reasonable exception to the
> rule.
> > > > >> Obviously users
> > > > >> should be able to detect the time when the record reaches the
> output
> > > > topic
> > > > >> but that
> > > > >> still leaves a gap in understanding how long the production
> latency
> > was.
> > > > >> This mirrors
> > > > >> the consumption latency that is exposed by the task-level metrics,
> > which
> > > > >> are measured
> > > > >> at the source node. For good symmetry what if we actually expose
> > both
> > > > the
> > > > >> source
> > > > >> and sink latency at the task-level? ie report both sets of
> > statistical
> > > > >> measurements with
> > > > >> the additional tag -source/-sink
> > > > >>
> > > > >> @Bill
> > > > >> Thanks for the comment about regarding the min! I hadn't
> considered
> > that
> > > > >> and it's
> > > > >> quite useful to think about how and what is useful from a users
> > point of
> > > > >> view.
> > > > >>
> > > > >> Regarding your second. point, I'm inclined to leave that as an
> > > > >> implementation detail
> > > > >> but my take would be that the user should be allowed to control
> the
> > > > record
> > > > >> timestamp
> > > > >> used for this with the timestamp extractor. My impression is that
> > users
> > > > may
> > > > >> often embed
> > > > >> the actual event time in the payload for whatever reason, and this
> > > > >> represents the "true"
> > > > >> timestamp as far as the Streams topology is concerned.
> > > > >>
> > > > >>
> > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com>
> > wrote:
> > > > >>
> > > > >>> Thanks for the KIP, Sophie, this will be a useful metric to add.
> > > > >>>
> > > > >>> Regarding tracking min, I  think it could be valuable for users
> to
> > > > >> discern
> > > > >>> which part of their topologies are more efficient since this is a
> > > > >>> task-level metric.  I realize everyone seems to be on board with
> > > > >> including
> > > > >>> min anyway, but I wanted to add my 2 cents on this topic should
> we
> > > > decide
> > > > >>> to revisit adding min or not.
> > > > >>>
> > > > >>> I do have a question regarding the calculation of staleness.
> > > > >>> Is there going to be a consideration for timestamp extractors?
> > Users
> > > > >> could
> > > > >>> prefer to use a timestamp embedded in the payload, and it could
> > skew
> > > > the
> > > > >>> measurements.
> > > > >>> I was wondering if we should specify in the KIP if setting the
> > arrival
> > > > >> time
> > > > >>> is always going to come from the record timestamp, or is this an
> > > > >>> implementation detail we can cover in the PR?
> > > > >>>
> > > > >>> Thanks!
> > > > >>> Bill
> > > > >>>
> > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <
> mjsax@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>>> Thanks for the KIP Sophie.
> > > > >>>>
> > > > >>>> I think it's not useful to record the avg/mean; it sensitive to
> > > > >>>> outliers. We should rather track the median (50th percentile).
> > > > >>>>
> > > > >>>> Not sure if tracking min is useful, but I am also ok to track
> it.
> > > > >>>>
> > > > >>>> However, I find it odd to track 75th percentile. Standard
> measures
> > > > >> would
> > > > >>>> the 90th or 95th -- I guess we don't need both, so maybe picking
> > 90th
> > > > >>>> might be more useful?
> > > > >>>>
> > > > >>>> About the name: "staleness" wound really odd, and if fact the
> > metric
> > > > >>>> does capture "latency" so we should call it "latency". I
> > understand
> > > > the
> > > > >>>> issue that we already have a latency metric. So maybe we could
> > call it
> > > > >>>> `record-e2e-latency-*` ?
> > > > >>>>
> > > > >>>> While I agree that we should include out-or-order data (the KIP
> > should
> > > > >>>> talk about `out-of-order` data, not `late` data; data is only
> > `late`
> > > > if
> > > > >>>> it's out-of-order and if it's dropped), I don't really
> understand
> > why
> > > > >>>> the new metric would help to configure grace period or retention
> > time?
> > > > >>>> As you mention in the KIP, both are define as max difference of
> > > > >>>> `event-time - stream-time` and thus the new metric that takes
> > > > >>>> system-/wallclock-time into account does not seem to help at
> all.
> > > > >>>>
> > > > >>>>
> > > > >>>> Btw: there is a great talk about "How NOT to Measure Latency" by
> > Gil
> > > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > > >>>>
> > > > >>>>
> > > > >>>> -Matthias
> > > > >>>>
> > > > >>>>
> > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > > >>>>> Hi Sophie,
> > > > >>>>>
> > > > >>>>> It seems like there would still be plenty of use cases for
> > recording
> > > > >>>>> this metric at all processors and not just stateful ones, but
> I'm
> > > > >> happy
> > > > >>>>> to suspend my arguments for now. Since you're proposing to keep
> > > > >>>>> them at the processor-node level, it will be seamless later to
> > add
> > > > >>>>> in the stateless processors if we want. As a wise man once
> said,
> > > > >>>>> "Adding is always easier than removing."
> > > > >>>>>
> > > > >>>>> Regarding the time measurement, it's an implementation detail
> > > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
> > > > >>>>> prefer to measure the system time again when recording the
> > > > >>>>> metric. I don't think we've seen any evidence that proves this
> > > > >>>>> would harm performance, and the amount of inaccuracy using
> > > > >>>>> the cached system time could incur is actually substantial.
> But,
> > > > >>>>> if you want to just "not mention this" in the KIP, we can defer
> > to
> > > > >>>>> the actual PR discussion, at which time we're in a better
> > position
> > > > >>>>> to use benchmarks, etc., to make the call.
> > > > >>>>>
> > > > >>>>> Along the lines of the measurement accuracy discussion, one
> > > > >>>>> minor thought I had is that maybe we should consider measuring
> > > > >>>>> the task staleness metric at the sink, rather than the source,
> so
> > > > >> that
> > > > >>>>> it includes the processing latency of the task itself, not just
> > the
> > > > >>>> latency
> > > > >>>>> of everything up to, but not including, the task (which seems
> > > > >> confusing
> > > > >>>>> for users). I guess this could also be an implementation
> detail,
> > > > >>> though.
> > > > >>>>>
> > > > >>>>> Thanks for the update,
> > > > >>>>> -John
> > > > >>>>>
> > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > > > >>>>>> Hey all,
> > > > >>>>>>
> > > > >>>>>> After discussing with Bruno I'd like to propose a small
> > amendment,
> > > > >>>>>> which is to record the processor-node-level metrics only for
> > > > >>> *stateful*
> > > > >>>>>> *operators*. They would still be considered a
> > "processor-node-level"
> > > > >>>>>> metric and not a "state-store-level" metric as the staleness
> is
> > > > >> still
> > > > >>>>>> a property of the node rather than of the state itself.
> > However, it
> > > > >>>> seems
> > > > >>>>>> that this information is primarily useful for stateful
> operators
> > > > >> that
> > > > >>>> might
> > > > >>>>>> be exposing state via IQ or otherwise dependent on the record
> > time
> > > > >>>>>> unlike a stateless operator.
> > > > >>>>>>
> > > > >>>>>> It's worth calling out that recent performance improvements to
> > the
> > > > >>>> metrics
> > > > >>>>>> framework mean that we no longer fetch the system time at the
> > > > >> operator
> > > > >>>>>> level, but only once per task. In other words the system time
> > is not
> > > > >>>> updated
> > > > >>>>>> between each process as a record flows through the
> subtopology,
> > so
> > > > >>>>>> debugging the processor-level latency via the stateleness will
> > not
> > > > >> be
> > > > >>>>>> possible.Note that this doesn't mean the operator-level
> metrics
> > are
> > > > >>> not
> > > > >>>>>> *useful* relative to the task-level metric. Upstream caching
> > and/or
> > > > >>>>>> suppression
> > > > >>>>>> can still cause a record's staleness at some downstream
> stateful
> > > > >>>> operator
> > > > >>>>>> to deviate from the task-level staleness (recorded at the
> source
> > > > >>> node).
> > > > >>>>>>
> > > > >>>>>> Please let me know if you have any concerns about this change.
> > The
> > > > >>>>>> KIP has been updated with the new proposal
> > > > >>>>>>
> > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> > bruno@confluent.io>
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Sophie,
> > > > >>>>>>>
> > > > >>>>>>> Thank you for the KIP.
> > > > >>>>>>>
> > > > >>>>>>> The KIP looks good to me.
> > > > >>>>>>>
> > > > >>>>>>> 50th percentile:
> > > > >>>>>>> I think we do not need it now. If we need it, we can add it.
> > Here
> > > > >> the
> > > > >>>>>>> old truism applies: Adding is always easier than removing.
> > > > >>>>>>>
> > > > >>>>>>> processor-node-level metrics:
> > > > >>>>>>> I think it is good to have the staleness metrics also on
> > > > >>>>>>> processor-node-level. If we do not want to record them on all
> > > > >>>>>>> processor nodes, you could restrict the recording to stateful
> > > > >>>>>>> processor-nodes, since those are the ones that would benefit
> > most
> > > > >>> from
> > > > >>>>>>> the staleness metrics.
> > > > >>>>>>>
> > > > >>>>>>> Best,
> > > > >>>>>>> Bruno
> > > > >>>>>>>
> > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > > >>>> sophie@confluent.io>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Yeah, the specific reason was just to align with the current
> > > > >>> metrics.
> > > > >>>>>>>>
> > > > >>>>>>>> Is it better to conform than to be right? History has a lot
> > to say
> > > > >>> on
> > > > >>>>>>> that
> > > > >>>>>>>> matter
> > > > >>>>>>>> but I'm not sure how much of it applies to the fine details
> of
> > > > >>> metrics
> > > > >>>>>>>> naming :P
> > > > >>>>>>>>
> > > > >>>>>>>> More seriously, I figured if people are looking at this
> metric
> > > > >>> they're
> > > > >>>>>>>> likely to
> > > > >>>>>>>> be looking at all the others. Then naming this one "-mean"
> > would
> > > > >>>> probably
> > > > >>>>>>>> lead some to conclude that the "-avg" suffix in the other
> > metrics
> > > > >>> has
> > > > >>>> a
> > > > >>>>>>>> different meaning.
> > > > >>>>>>>>
> > > > >>>>>>>> As for the percentiles, I actually like p99 (and p75)
> better.
> > I'll
> > > > >>>> swap
> > > > >>>>>>>> that out
> > > > >>>>>>>>
> > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> > vvcephei@apache.org
> > > > >>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Thanks Sophie,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I hope this isn't too nit-picky, but is there a reason to
> > choose
> > > > >>>> "avg"
> > > > >>>>>>>>> instead
> > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > > > >> oversensitive
> > > > >>>>>>> because
> > > > >>>>>>>>> of the mistake I just made earlier, but it strikes me that
> > "avg"
> > > > >> is
> > > > >>>>>>>>> actually
> > > > >>>>>>>>> ambiguous, as it refers to a family of statistics, whereas
> > "mean"
> > > > >>> is
> > > > >>>>>>>>> specific.
> > > > >>>>>>>>> I see other Kafka metrics with "avg", but none with "mean";
> > was
> > > > >>> that
> > > > >>>>>>> the
> > > > >>>>>>>>> reason? If so, I'm +1.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Regarding the names of the percentile, I actually couldn't
> > find
> > > > >>> _any_
> > > > >>>>>>> other
> > > > >>>>>>>>> metrics that use percentile. Was there a reason to choose
> > "99th"
> > > > >> as
> > > > >>>>>>> opposed
> > > > >>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm
> > just
> > > > >>>>>>> primarily
> > > > >>>>>>>>> asking
> > > > >>>>>>>>> for consistency's sake.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks again,
> > > > >>>>>>>>> -John
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > > >>>>>>>>>> Alright, I can get behind adding the min metric for the
> > sake of
> > > > >>>>>>> pretty
> > > > >>>>>>>>>> graphs
> > > > >>>>>>>>>> (and trivial computation).
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> > percentile)
> > > > >>> but I
> > > > >>>>>>> can
> > > > >>>>>>>>> see
> > > > >>>>>>>>>> how users might expect it and find it a bit disorienting
> > not to
> > > > >>>>>>> have. So
> > > > >>>>>>>>> the
> > > > >>>>>>>>>> updated proposed metrics are
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>    - record-staleness-max [ms]
> > > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > > >>>>>>>>>>    - record-staleness-min [ms]
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > > >> vvcephei@apache.org
> > > > >>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself.
> I
> > > > >> guess
> > > > >>>>>>> the
> > > > >>>>>>>>> mean
> > > > >>>>>>>>>>> seems more interesting to me than the median, but neither
> > are
> > > > >> as
> > > > >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it does
> round
> > out
> > > > >>> the
> > > > >>>>>>>>> mental
> > > > >>>>>>>>>>> picture of the distribution. I’ve always graphed min
> along
> > with
> > > > >>> the
> > > > >>>>>>>>> other
> > > > >>>>>>>>>>> metrics to help me understand how fast the system can be,
> > which
> > > > >>>>>>> helps
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>> optimization decisions. It’s also a relatively
> inexpensive
> > > > >> metric
> > > > >>>>>>> to
> > > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman
> wrote:
> > > > >>>>>>>>>>>> G1:
> > > > >>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to
> > the
> > > > >>>>>>> specific
> > > > >>>>>>>>>>> task"
> > > > >>>>>>>>>>>> but
> > > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the
> point
> > > > >> home
> > > > >>>>>>>>> better.
> > > > >>>>>>>>>>> So
> > > > >>>>>>>>>>>> it's the
> > > > >>>>>>>>>>>> "staleness of the record when it is received by that
> > task" --
> > > > >>>>>>> will
> > > > >>>>>>>>> update
> > > > >>>>>>>>>>>> the KIP
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> B1/J:
> > > > >>>>>>>>>>>> I'm struggling to imagine a case where the min would
> > actually
> > > > >> be
> > > > >>>>>>>>> useful,
> > > > >>>>>>>>>>>> rather than
> > > > >>>>>>>>>>>> just intellectually interesting. I don't feel strongly
> > that we
> > > > >>>>>>>>> shouldn't
> > > > >>>>>>>>>>>> add it, but that's
> > > > >>>>>>>>>>>> why I didn't include it from the start. Can you
> enlighten
> > me
> > > > >>>>>>> with an
> > > > >>>>>>>>>>>> example?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I was also vaguely concerned about the overhead of
> adding
> > > > >>>>>>> multiple
> > > > >>>>>>>>>>>> percentile
> > > > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
> > > > >>> performance
> > > > >>>>>>>>> hit we
> > > > >>>>>>>>>>>> take on
> > > > >>>>>>>>>>>> metrics computation?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile
> > would be
> > > > >>> the
> > > > >>>>>>>>> median
> > > > >>>>>>>>>>>> not the
> > > > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > > > >>>>>>> percentile, or
> > > > >>>>>>>>>>> just
> > > > >>>>>>>>>>>> one
> > > > >>>>>>>>>>>> of the two?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks all!
> > > > >>>>>>>>>>>> Sophie
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > > >>>>>>> vvcephei@apache.org>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> B2/G1:
> > > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we report
> > "spans"
> > > > >> or
> > > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the
> spans
> > are
> > > > >>>>>>> pretty
> > > > >>>>>>>>>>>>> difficult to really measure (like time spent waiting in
> > the
> > > > >>>>>>>>> topics, or
> > > > >>>>>>>>>>>>> time from the event happening to the ETL producer
> > choosing to
> > > > >>>>>>> send
> > > > >>>>>>>>> it,
> > > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> In other words, it's practically easier to compute
> spans
> > by
> > > > >>>>>>>>> subtracting
> > > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by
> > adding
> > > > >>>>>>> spans.
> > > > >>>>>>>>> You
> > > > >>>>>>>>>>>>> can even just consider that the span computation from
> e2e
> > > > >>>>>>> always
> > > > >>>>>>>>> just
> > > > >>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
> > > > >> latency
> > > > >>>>>>>>> from
> > > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the
> > end
> > > > >> you
> > > > >>>>>>>>> care
> > > > >>>>>>>>>>> about.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> It seems like people really prefer to have spans when
> > they
> > > > >> are
> > > > >>>>>>>>>>> debugging
> > > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
> > > > >>>>>>> measurement
> > > > >>>>>>>>>>>>> that basically every person/application cares about and
> > > > >> should
> > > > >>>>>>> be
> > > > >>>>>>>>>>>>> monitoring.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Altogether, it really seem to provide more value to
> more
> > > > >>>>>>> people if
> > > > >>>>>>>>> we
> > > > >>>>>>>>>>>>> report
> > > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name,
> I
> > > > >> think
> > > > >>>>>>> I
> > > > >>>>>>>>> have
> > > > >>>>>>>>>>> no
> > > > >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> G2:
> > > > >>>>>>>>>>>>> I think the processor-node metric is nice, since the
> > inside
> > > > >> of
> > > > >>>>>>> a
> > > > >>>>>>>>> task
> > > > >>>>>>>>>>> can
> > > > >>>>>>>>>>>>> introduce a significant amount of latency in some
> cases.
> > > > >> Plus,
> > > > >>>>>>>>> it's a
> > > > >>>>>>>>>>> more
> > > > >>>>>>>>>>>>> direct measurement, if you really wanted to know (for
> the
> > > > >>>>>>> purposes
> > > > >>>>>>>>> of
> > > > >>>>>>>>>>> IQ
> > > > >>>>>>>>>>>>> or something) how long it takes source events to "show
> > up" at
> > > > >>>>>>> the
> > > > >>>>>>>>>>> store.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I think actually recording it at every processor could
> be
> > > > >>>>>>>>> expensive,
> > > > >>>>>>>>>>> but we
> > > > >>>>>>>>>>>>> already record a bunch of metrics at the node level.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> B1:
> > > > >>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if
> > it's
> > > > >> a
> > > > >>>>>>> poor
> > > > >>>>>>>>>>> metric
> > > > >>>>>>>>>>>>> for operational purposes, a lot of people might expect
> > to see
> > > > >>>>>>>>> "mean".
> > > > >>>>>>>>>>>>> Actually,
> > > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to
> > leave
> > > > >> it
> > > > >>>>>>>>> off?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I might suggest:
> > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> B3:
> > > > >>>>>>>>>>>>> I agree we should include late records (though not the
> > ones
> > > > >> we
> > > > >>>>>>>>> drop).
> > > > >>>>>>>>>>>>> It may be spiky, but only when there are legitimately
> > some
> > > > >>>>>>> records
> > > > >>>>>>>>>>> with a
> > > > >>>>>>>>>>>>> high end-to-end latency, which is the whole point of
> > these
> > > > >>>>>>> metrics.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> That's it! I don't think I have any other feedback,
> other
> > > > >> than
> > > > >>>>>>> a
> > > > >>>>>>>>>>> request to
> > > > >>>>>>>>>>>>> also report "min".
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>> -John
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
> > > > >>>>>>> latency
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>> task
> > > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate topics.
> I
> > have
> > > > >>>>>>> a
> > > > >>>>>>>>>>> similar
> > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency
> > metric of
> > > > >>>>>>> a
> > > > >>>>>>>>> task A
> > > > >>>>>>>>>>>>>> actually measures the latency of the sub-topology
> up-to
> > but
> > > > >>>>>>> not
> > > > >>>>>>>>>>> including
> > > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency"
> > itself,
> > > > >>>>>>> since
> > > > >>>>>>>>>>> today we
> > > > >>>>>>>>>>>>>> already have several "latency" metrics already which
> are
> > > > >>>>>>>>> measuring
> > > > >>>>>>>>>>>>> elapsed
> > > > >>>>>>>>>>>>>> system-time for processing a record / etc, while here
> > we are
> > > > >>>>>>>>>>> comparing
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Maybe we can consider renaming it as
> "record-staleness"
> > > > >>>>>>> (note we
> > > > >>>>>>>>>>> already
> > > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case
> > recording at
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>>> system-time before we start processing the record
> sounds
> > > > >> more
> > > > >>>>>>>>>>> natural.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > > >>>>>>> processor-node-level
> > > > >>>>>>>>> DEBUG
> > > > >>>>>>>>>>>>>> metric is worth to add, given that we already have a
> > > > >>>>>>> task-level
> > > > >>>>>>>>>>>>> processing
> > > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e
> > latency is
> > > > >>>>>>>>> similar
> > > > >>>>>>>>>>> to
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing
> latency.
> > > > >>>>>>>>> Personally I
> > > > >>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> > sufficient.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Guozhang
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > >>>>>>>>>>>>> sophie@confluent.io>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful
> gauge
> > for
> > > > >>>>>>> this
> > > > >>>>>>>>>>>>> specific
> > > > >>>>>>>>>>>>>>> metric, as
> > > > >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper
> > *bound on
> > > > >>>>>>> the
> > > > >>>>>>>>>>> latency
> > > > >>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that
> > would
> > > > >>>>>>>>> hopefully
> > > > >>>>>>>>>>>>> give a
> > > > >>>>>>>>>>>>>>> good
> > > > >>>>>>>>>>>>>>> sense of what *most* records will experience, and
> what
> > > > >>>>>>> *close
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>> all*
> > > > >>>>>>>>>>>>>>> records
> > > > >>>>>>>>>>>>>>> will.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> However I'm not married to these specific numbers and
> > > > >>>>>>> could be
> > > > >>>>>>>>>>>>> convinced.
> > > > >>>>>>>>>>>>>>> Would be especially interested in hearing from users
> on
> > > > >>>>>>> this.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop
> > latency" in
> > > > >>>>>>>>> this KIP
> > > > >>>>>>>>>>>>> since
> > > > >>>>>>>>>>>>>>> users
> > > > >>>>>>>>>>>>>>> can always compute it themselves by subtracting the
> > > > >>>>>>> previous
> > > > >>>>>>>>> node's
> > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way
> > since
> > > > >>>>>>>>> you can
> > > > >>>>>>>>>>>>> always
> > > > >>>>>>>>>>>>>>> compute one from the other, but I think the
> end-to-end
> > > > >>>>>>> latency
> > > > >>>>>>>>>>> feels
> > > > >>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> > > > >>>>>>> bottlenecks
> > > > >>>>>>>>> in the
> > > > >>>>>>>>>>>>>>> topology but
> > > > >>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to
> > be
> > > > >>>>>>>>> reflected
> > > > >>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>> certain parts
> > > > >>>>>>>>>>>>>>> of the topology. For example this might be useful for
> > users
> > > > >>>>>>>>> who are
> > > > >>>>>>>>>>>>>>> wondering
> > > > >>>>>>>>>>>>>>> roughly when a record that was just produced will be
> > > > >>>>>>> included
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>> their
> > > > >>>>>>>>>>>>> IQ
> > > > >>>>>>>>>>>>>>> results.
> > > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I
> > didn't
> > > > >>>>>>> make
> > > > >>>>>>>>>>> that
> > > > >>>>>>>>>>>>> clear
> > > > >>>>>>>>>>>>>>> enough
> > > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP.
> The
> > > > >>>>>>> short
> > > > >>>>>>>>>>> answer is
> > > > >>>>>>>>>>>>>>> "yes",
> > > > >>>>>>>>>>>>>>> we will include late records. I added a paragraph to
> > the
> > > > >>>>>>> end
> > > > >>>>>>>>> of the
> > > > >>>>>>>>>>>>>>> Proposed
> > > > >>>>>>>>>>>>>>> Changes section explaining the reasoning here, please
> > let
> > > > >>>>>>> me
> > > > >>>>>>>>> know
> > > > >>>>>>>>>>> if
> > > > >>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>> any concerns.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > > > >>>>>>>>>>> "process-latency",
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>> metric
> > > > >>>>>>>>>>>>>>> reflects the time for the literal Node#process method
> > to
> > > > >>>>>>> run
> > > > >>>>>>>>>>> whereas
> > > > >>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>> metric
> > > > >>>>>>>>>>>>>>> would always be measured relative to the event
> > timestamp.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> That said, the naming collision there is pretty
> > confusing
> > > > >>>>>>> so
> > > > >>>>>>>>> I've
> > > > >>>>>>>>>>>>> renamed
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I
> > feel
> > > > >>>>>>> better
> > > > >>>>>>>>>>>>> reflects
> > > > >>>>>>>>>>>>>>> the nature
> > > > >>>>>>>>>>>>>>> of the metric anyway.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency
> is
> > > > >>>>>>>>> important
> > > > >>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99
> > and
> > > > >>>>>>> max
> > > > >>>>>>>>>>>>> percentiles?
> > > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile
> as
> > > > >>>>>>> well
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>>>> production
> > > > >>>>>>>>>>>>>>>> systems.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative,
> > I.E
> > > > >>>>>>> if a
> > > > >>>>>>>>>>> record
> > > > >>>>>>>>>>>>> goes
> > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > > > >>>>>>> T(B->C) +
> > > > >>>>>>>>>>> T(A->B) +
> > > > >>>>>>>>>>>>>>> T(A)
> > > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured
> latency,
> > > > >>>>>>> and T()
> > > > >>>>>>>>>>>>> represents
> > > > >>>>>>>>>>>>>>>> the time for transiting the records between two
> nodes,
> > > > >>>>>>>>> including
> > > > >>>>>>>>>>>>>>> processing
> > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C)
> and
> > > > >>>>>>>>> T(A->B)
> > > > >>>>>>>>>>> are
> > > > >>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise
> if
> > > > >>>>>>> there
> > > > >>>>>>>>> is a
> > > > >>>>>>>>>>>>> spike in
> > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> > > > >>>>>>> time.
> > > > >>>>>>>>> In the
> > > > >>>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> > > > >>>>>>> record
> > > > >>>>>>>>> exits
> > > > >>>>>>>>>>>>> from the
> > > > >>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
> > > > >>>>>>> inside
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>> funnel. Do
> > > > >>>>>>>>>>>>>>>> you think we could have separate treatment for sink
> > > > >>>>>>> nodes and
> > > > >>>>>>>>>>> other
> > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time
> > receiving
> > > > >>>>>>> the
> > > > >>>>>>>>>>> record
> > > > >>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just
> > want to
> > > > >>>>>>>>> discuss
> > > > >>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records
> as
> > > > >>>>>>> well,
> > > > >>>>>>>>> they
> > > > >>>>>>>>>>>>> would
> > > > >>>>>>>>>>>>>>>> create some really spiky graphs when the
> out-of-order
> > > > >>>>>>>>> records are
> > > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we also
> > supply
> > > > >>>>>>> a
> > > > >>>>>>>>> smooth
> > > > >>>>>>>>>>>>> version
> > > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just take
> care
> > of
> > > > >>>>>>> it
> > > > >>>>>>>>> by
> > > > >>>>>>>>>>>>> themself?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed
> > its
> > > > >>>>>>>>> relation
> > > > >>>>>>>>>>>>> with our
> > > > >>>>>>>>>>>>>>>> existing processing latency metrics, could you add
> > some
> > > > >>>>>>>>> context
> > > > >>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial
> > for
> > > > >>>>>>> the
> > > > >>>>>>>>>>> best?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Boyang
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman
> <
> > > > >>>>>>>>>>>>> sophie@confluent.io
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hey all,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which
> aims
> > > > >>>>>>> to
> > > > >>>>>>>>> add
> > > > >>>>>>>>>>>>> end-to-end
> > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>> Sophie
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>> -- Guozhang
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
> @Matthias
> And that's what he says about the 95th percentile! Imagine what he would
say about the 50th :P

I should have kept watching another 10 minutes. He gets around to covering
the 50th, and let's
just say he is not a fan: https://youtu.be/lJ8ydIuPFeU?t=786

On Fri, May 15, 2020 at 3:15 PM John Roesler <vv...@apache.org> wrote:

> Thanks, Sophie!
>
> I think this makes perfect sense. It will be much more intuitive to use
> the metric for the stated motivation this way.  I’d be in favor of the
> proposal after this update.
>
> Thanks again for taking this on,
> -John
>
> On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> > Hi all,
> >
> > I'd like to clarify/modify one aspect of this KIP, which is to record the
> > staleness
> > at the *completion* of the record's processing by the operator/task in
> > question,
> > rather than on the intake. The task-level metrics will be recorded at the
> > sink
> > node instead of at the source, and the operator-level metrics will be
> > recorded
> > at the end of the operation.
> >
> > The stated purpose and intended usefulness of this KIP is to give users a
> > way
> > to gauge roughly how long it takes for a record to be reflected in the
> > "results",
> > whether these results are being read from an output topic or through IQ.
> To
> > take the IQ example, the results of a record are obviously not visible
> until
> > *after* that node has finished processing it. The staleness, while still
> > potentially
> > useful as it can impact the *way* a record is processed in a stateful and
> > time-
> > dependent operator, is not part of the problem this KIP specifically set
> out
> > to solve.
> >
> > In light of this I think it's appropriate to revert the name change back
> to
> > include
> > latency, since "staleness" as described above only makes sense when
> > measuring
> > relative to the arrival of a record at a task/node. I'd propose to save
> the
> > term
> > "staleness" for that particular meaning, and adopt Matthias's suggestion
> of
> > "record-e2e-latency" for this.
> >
> > Thanks for hanging in there all. Please let me know if you have any
> > concerns about
> > this change!
> >
> > Sophie
> >
> > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > I am also happy with max/min/99/90. And I buy your naming argument
> about
> > > staleness vs latency.
> > >
> > >
> > > -Matthias
> > >
> > > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > > Hey Sophie,
> > > >
> > > > 90/99/min/max make sense to me.
> > > >
> > > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > > sophie@confluent.io>
> > > > wrote:
> > > >
> > > >> @Matthias
> > > >> Regarding tracking the 50th percentile, I'll refer you to the 4:53
> mark
> > > of
> > > >> the video
> > > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > > >>
> > > >> And that's what he says about the 95th percentile! Imagine what he
> would
> > > >> say about
> > > >> the 50th :P
> > > >>
> > > >> But seriously, since we can't seem to agree that the mean or 50th
> > > >> percentile is actually
> > > >> useful I'm inclined to resurrect my original proposal, neither. But
> I
> > > think
> > > >> that's a good
> > > >> argument against the 75th, which I admittedly chose somewhat
> > > arbitrarily as
> > > >> an
> > > >> intermediate between the 50th and the higher percentiles. How about:
> > > >>
> > > >> -max
> > > >> -p99
> > > >> -p90
> > > >> -min
> > > >>
> > > >> with p50/mean still up for debate if anyone feels strongly for
> either of
> > > >> them.
> > > >>
> > > >> Regarding the name, I've already flip-flopped on this so I'm
> definitely
> > > >> still open to
> > > >> further arguments. But the reason for changing it from
> > > end-to-end-latency
> > > >> (which
> > > >> is similar to what you propose) is that this metric technically
> reflects
> > > >> how old (ie how "stale")
> > > >> the record is when it's *received* by the operator, not when it's
> > > processed
> > > >> by the operator.
> > > >> It seemed like there was the potential for confusion that
> > > >> "end-to-end-latency" might
> > > >> represent the latency from the event creation to the time the
> processor
> > > is
> > > >> done
> > > >> processing it.
> > > >>
> > > >> @John
> > > >> I'd rather err on the side of "not-enough" metrics as we can always
> add
> > > >> this to the
> > > >> stateless metrics later on. If we decide to measure the time at
> every
> > > node
> > > >> and don't
> > > >> find any evidence of a serious performance impact, and users
> indicate
> > > they
> > > >> would
> > > >> like to see this metric at all nodes, then we can easily start
> reporting
> > > >> them as well.
> > > >> WDYT?
> > > >>
> > > >> That said, sink nodes seem like a reasonable exception to the rule.
> > > >> Obviously users
> > > >> should be able to detect the time when the record reaches the output
> > > topic
> > > >> but that
> > > >> still leaves a gap in understanding how long the production latency
> was.
> > > >> This mirrors
> > > >> the consumption latency that is exposed by the task-level metrics,
> which
> > > >> are measured
> > > >> at the source node. For good symmetry what if we actually expose
> both
> > > the
> > > >> source
> > > >> and sink latency at the task-level? ie report both sets of
> statistical
> > > >> measurements with
> > > >> the additional tag -source/-sink
> > > >>
> > > >> @Bill
> > > >> Thanks for the comment about regarding the min! I hadn't considered
> that
> > > >> and it's
> > > >> quite useful to think about how and what is useful from a users
> point of
> > > >> view.
> > > >>
> > > >> Regarding your second. point, I'm inclined to leave that as an
> > > >> implementation detail
> > > >> but my take would be that the user should be allowed to control the
> > > record
> > > >> timestamp
> > > >> used for this with the timestamp extractor. My impression is that
> users
> > > may
> > > >> often embed
> > > >> the actual event time in the payload for whatever reason, and this
> > > >> represents the "true"
> > > >> timestamp as far as the Streams topology is concerned.
> > > >>
> > > >>
> > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com>
> wrote:
> > > >>
> > > >>> Thanks for the KIP, Sophie, this will be a useful metric to add.
> > > >>>
> > > >>> Regarding tracking min, I  think it could be valuable for users to
> > > >> discern
> > > >>> which part of their topologies are more efficient since this is a
> > > >>> task-level metric.  I realize everyone seems to be on board with
> > > >> including
> > > >>> min anyway, but I wanted to add my 2 cents on this topic should we
> > > decide
> > > >>> to revisit adding min or not.
> > > >>>
> > > >>> I do have a question regarding the calculation of staleness.
> > > >>> Is there going to be a consideration for timestamp extractors?
> Users
> > > >> could
> > > >>> prefer to use a timestamp embedded in the payload, and it could
> skew
> > > the
> > > >>> measurements.
> > > >>> I was wondering if we should specify in the KIP if setting the
> arrival
> > > >> time
> > > >>> is always going to come from the record timestamp, or is this an
> > > >>> implementation detail we can cover in the PR?
> > > >>>
> > > >>> Thanks!
> > > >>> Bill
> > > >>>
> > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
> > > >> wrote:
> > > >>>
> > > >>>> Thanks for the KIP Sophie.
> > > >>>>
> > > >>>> I think it's not useful to record the avg/mean; it sensitive to
> > > >>>> outliers. We should rather track the median (50th percentile).
> > > >>>>
> > > >>>> Not sure if tracking min is useful, but I am also ok to track it.
> > > >>>>
> > > >>>> However, I find it odd to track 75th percentile. Standard measures
> > > >> would
> > > >>>> the 90th or 95th -- I guess we don't need both, so maybe picking
> 90th
> > > >>>> might be more useful?
> > > >>>>
> > > >>>> About the name: "staleness" wound really odd, and if fact the
> metric
> > > >>>> does capture "latency" so we should call it "latency". I
> understand
> > > the
> > > >>>> issue that we already have a latency metric. So maybe we could
> call it
> > > >>>> `record-e2e-latency-*` ?
> > > >>>>
> > > >>>> While I agree that we should include out-or-order data (the KIP
> should
> > > >>>> talk about `out-of-order` data, not `late` data; data is only
> `late`
> > > if
> > > >>>> it's out-of-order and if it's dropped), I don't really understand
> why
> > > >>>> the new metric would help to configure grace period or retention
> time?
> > > >>>> As you mention in the KIP, both are define as max difference of
> > > >>>> `event-time - stream-time` and thus the new metric that takes
> > > >>>> system-/wallclock-time into account does not seem to help at all.
> > > >>>>
> > > >>>>
> > > >>>> Btw: there is a great talk about "How NOT to Measure Latency" by
> Gil
> > > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > > >>>>
> > > >>>>
> > > >>>> -Matthias
> > > >>>>
> > > >>>>
> > > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > > >>>>> Hi Sophie,
> > > >>>>>
> > > >>>>> It seems like there would still be plenty of use cases for
> recording
> > > >>>>> this metric at all processors and not just stateful ones, but I'm
> > > >> happy
> > > >>>>> to suspend my arguments for now. Since you're proposing to keep
> > > >>>>> them at the processor-node level, it will be seamless later to
> add
> > > >>>>> in the stateless processors if we want. As a wise man once said,
> > > >>>>> "Adding is always easier than removing."
> > > >>>>>
> > > >>>>> Regarding the time measurement, it's an implementation detail
> > > >>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
> > > >>>>> prefer to measure the system time again when recording the
> > > >>>>> metric. I don't think we've seen any evidence that proves this
> > > >>>>> would harm performance, and the amount of inaccuracy using
> > > >>>>> the cached system time could incur is actually substantial. But,
> > > >>>>> if you want to just "not mention this" in the KIP, we can defer
> to
> > > >>>>> the actual PR discussion, at which time we're in a better
> position
> > > >>>>> to use benchmarks, etc., to make the call.
> > > >>>>>
> > > >>>>> Along the lines of the measurement accuracy discussion, one
> > > >>>>> minor thought I had is that maybe we should consider measuring
> > > >>>>> the task staleness metric at the sink, rather than the source, so
> > > >> that
> > > >>>>> it includes the processing latency of the task itself, not just
> the
> > > >>>> latency
> > > >>>>> of everything up to, but not including, the task (which seems
> > > >> confusing
> > > >>>>> for users). I guess this could also be an implementation detail,
> > > >>> though.
> > > >>>>>
> > > >>>>> Thanks for the update,
> > > >>>>> -John
> > > >>>>>
> > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > > >>>>>> Hey all,
> > > >>>>>>
> > > >>>>>> After discussing with Bruno I'd like to propose a small
> amendment,
> > > >>>>>> which is to record the processor-node-level metrics only for
> > > >>> *stateful*
> > > >>>>>> *operators*. They would still be considered a
> "processor-node-level"
> > > >>>>>> metric and not a "state-store-level" metric as the staleness is
> > > >> still
> > > >>>>>> a property of the node rather than of the state itself.
> However, it
> > > >>>> seems
> > > >>>>>> that this information is primarily useful for stateful operators
> > > >> that
> > > >>>> might
> > > >>>>>> be exposing state via IQ or otherwise dependent on the record
> time
> > > >>>>>> unlike a stateless operator.
> > > >>>>>>
> > > >>>>>> It's worth calling out that recent performance improvements to
> the
> > > >>>> metrics
> > > >>>>>> framework mean that we no longer fetch the system time at the
> > > >> operator
> > > >>>>>> level, but only once per task. In other words the system time
> is not
> > > >>>> updated
> > > >>>>>> between each process as a record flows through the subtopology,
> so
> > > >>>>>> debugging the processor-level latency via the stateleness will
> not
> > > >> be
> > > >>>>>> possible.Note that this doesn't mean the operator-level metrics
> are
> > > >>> not
> > > >>>>>> *useful* relative to the task-level metric. Upstream caching
> and/or
> > > >>>>>> suppression
> > > >>>>>> can still cause a record's staleness at some downstream stateful
> > > >>>> operator
> > > >>>>>> to deviate from the task-level staleness (recorded at the source
> > > >>> node).
> > > >>>>>>
> > > >>>>>> Please let me know if you have any concerns about this change.
> The
> > > >>>>>> KIP has been updated with the new proposal
> > > >>>>>>
> > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <
> bruno@confluent.io>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Sophie,
> > > >>>>>>>
> > > >>>>>>> Thank you for the KIP.
> > > >>>>>>>
> > > >>>>>>> The KIP looks good to me.
> > > >>>>>>>
> > > >>>>>>> 50th percentile:
> > > >>>>>>> I think we do not need it now. If we need it, we can add it.
> Here
> > > >> the
> > > >>>>>>> old truism applies: Adding is always easier than removing.
> > > >>>>>>>
> > > >>>>>>> processor-node-level metrics:
> > > >>>>>>> I think it is good to have the staleness metrics also on
> > > >>>>>>> processor-node-level. If we do not want to record them on all
> > > >>>>>>> processor nodes, you could restrict the recording to stateful
> > > >>>>>>> processor-nodes, since those are the ones that would benefit
> most
> > > >>> from
> > > >>>>>>> the staleness metrics.
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Bruno
> > > >>>>>>>
> > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > >>>> sophie@confluent.io>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Yeah, the specific reason was just to align with the current
> > > >>> metrics.
> > > >>>>>>>>
> > > >>>>>>>> Is it better to conform than to be right? History has a lot
> to say
> > > >>> on
> > > >>>>>>> that
> > > >>>>>>>> matter
> > > >>>>>>>> but I'm not sure how much of it applies to the fine details of
> > > >>> metrics
> > > >>>>>>>> naming :P
> > > >>>>>>>>
> > > >>>>>>>> More seriously, I figured if people are looking at this metric
> > > >>> they're
> > > >>>>>>>> likely to
> > > >>>>>>>> be looking at all the others. Then naming this one "-mean"
> would
> > > >>>> probably
> > > >>>>>>>> lead some to conclude that the "-avg" suffix in the other
> metrics
> > > >>> has
> > > >>>> a
> > > >>>>>>>> different meaning.
> > > >>>>>>>>
> > > >>>>>>>> As for the percentiles, I actually like p99 (and p75) better.
> I'll
> > > >>>> swap
> > > >>>>>>>> that out
> > > >>>>>>>>
> > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <
> vvcephei@apache.org
> > > >>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks Sophie,
> > > >>>>>>>>>
> > > >>>>>>>>> I hope this isn't too nit-picky, but is there a reason to
> choose
> > > >>>> "avg"
> > > >>>>>>>>> instead
> > > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > > >> oversensitive
> > > >>>>>>> because
> > > >>>>>>>>> of the mistake I just made earlier, but it strikes me that
> "avg"
> > > >> is
> > > >>>>>>>>> actually
> > > >>>>>>>>> ambiguous, as it refers to a family of statistics, whereas
> "mean"
> > > >>> is
> > > >>>>>>>>> specific.
> > > >>>>>>>>> I see other Kafka metrics with "avg", but none with "mean";
> was
> > > >>> that
> > > >>>>>>> the
> > > >>>>>>>>> reason? If so, I'm +1.
> > > >>>>>>>>>
> > > >>>>>>>>> Regarding the names of the percentile, I actually couldn't
> find
> > > >>> _any_
> > > >>>>>>> other
> > > >>>>>>>>> metrics that use percentile. Was there a reason to choose
> "99th"
> > > >> as
> > > >>>>>>> opposed
> > > >>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm
> just
> > > >>>>>>> primarily
> > > >>>>>>>>> asking
> > > >>>>>>>>> for consistency's sake.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks again,
> > > >>>>>>>>> -John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > >>>>>>>>>> Alright, I can get behind adding the min metric for the
> sake of
> > > >>>>>>> pretty
> > > >>>>>>>>>> graphs
> > > >>>>>>>>>> (and trivial computation).
> > > >>>>>>>>>>
> > > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th
> percentile)
> > > >>> but I
> > > >>>>>>> can
> > > >>>>>>>>> see
> > > >>>>>>>>>> how users might expect it and find it a bit disorienting
> not to
> > > >>>>>>> have. So
> > > >>>>>>>>> the
> > > >>>>>>>>>> updated proposed metrics are
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>    - record-staleness-max [ms]
> > > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > > >>>>>>>>>>    - record-staleness-min [ms]
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > > >> vvcephei@apache.org
> > > >>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
> > > >> guess
> > > >>>>>>> the
> > > >>>>>>>>> mean
> > > >>>>>>>>>>> seems more interesting to me than the median, but neither
> are
> > > >> as
> > > >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Min isn’t really important for any SLAs, but it does round
> out
> > > >>> the
> > > >>>>>>>>> mental
> > > >>>>>>>>>>> picture of the distribution. I’ve always graphed min along
> with
> > > >>> the
> > > >>>>>>>>> other
> > > >>>>>>>>>>> metrics to help me understand how fast the system can be,
> which
> > > >>>>>>> helps
> > > >>>>>>>>> in
> > > >>>>>>>>>>> optimization decisions. It’s also a relatively inexpensive
> > > >> metric
> > > >>>>>>> to
> > > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > >>>>>>>>>>>> G1:
> > > >>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to
> the
> > > >>>>>>> specific
> > > >>>>>>>>>>> task"
> > > >>>>>>>>>>>> but
> > > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the point
> > > >> home
> > > >>>>>>>>> better.
> > > >>>>>>>>>>> So
> > > >>>>>>>>>>>> it's the
> > > >>>>>>>>>>>> "staleness of the record when it is received by that
> task" --
> > > >>>>>>> will
> > > >>>>>>>>> update
> > > >>>>>>>>>>>> the KIP
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> B1/J:
> > > >>>>>>>>>>>> I'm struggling to imagine a case where the min would
> actually
> > > >> be
> > > >>>>>>>>> useful,
> > > >>>>>>>>>>>> rather than
> > > >>>>>>>>>>>> just intellectually interesting. I don't feel strongly
> that we
> > > >>>>>>>>> shouldn't
> > > >>>>>>>>>>>> add it, but that's
> > > >>>>>>>>>>>> why I didn't include it from the start. Can you enlighten
> me
> > > >>>>>>> with an
> > > >>>>>>>>>>>> example?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I was also vaguely concerned about the overhead of adding
> > > >>>>>>> multiple
> > > >>>>>>>>>>>> percentile
> > > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
> > > >>> performance
> > > >>>>>>>>> hit we
> > > >>>>>>>>>>>> take on
> > > >>>>>>>>>>>> metrics computation?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile
> would be
> > > >>> the
> > > >>>>>>>>> median
> > > >>>>>>>>>>>> not the
> > > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > > >>>>>>> percentile, or
> > > >>>>>>>>>>> just
> > > >>>>>>>>>>>> one
> > > >>>>>>>>>>>> of the two?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks all!
> > > >>>>>>>>>>>> Sophie
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > >>>>>>> vvcephei@apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> B2/G1:
> > > >>>>>>>>>>>>> In principle, it shouldn't matter whether we report
> "spans"
> > > >> or
> > > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the spans
> are
> > > >>>>>>> pretty
> > > >>>>>>>>>>>>> difficult to really measure (like time spent waiting in
> the
> > > >>>>>>>>> topics, or
> > > >>>>>>>>>>>>> time from the event happening to the ETL producer
> choosing to
> > > >>>>>>> send
> > > >>>>>>>>> it,
> > > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> In other words, it's practically easier to compute spans
> by
> > > >>>>>>>>> subtracting
> > > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by
> adding
> > > >>>>>>> spans.
> > > >>>>>>>>> You
> > > >>>>>>>>>>>>> can even just consider that the span computation from e2e
> > > >>>>>>> always
> > > >>>>>>>>> just
> > > >>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
> > > >> latency
> > > >>>>>>>>> from
> > > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the
> end
> > > >> you
> > > >>>>>>>>> care
> > > >>>>>>>>>>> about.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> It seems like people really prefer to have spans when
> they
> > > >> are
> > > >>>>>>>>>>> debugging
> > > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
> > > >>>>>>> measurement
> > > >>>>>>>>>>>>> that basically every person/application cares about and
> > > >> should
> > > >>>>>>> be
> > > >>>>>>>>>>>>> monitoring.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Altogether, it really seem to provide more value to more
> > > >>>>>>> people if
> > > >>>>>>>>> we
> > > >>>>>>>>>>>>> report
> > > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
> > > >> think
> > > >>>>>>> I
> > > >>>>>>>>> have
> > > >>>>>>>>>>> no
> > > >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> G2:
> > > >>>>>>>>>>>>> I think the processor-node metric is nice, since the
> inside
> > > >> of
> > > >>>>>>> a
> > > >>>>>>>>> task
> > > >>>>>>>>>>> can
> > > >>>>>>>>>>>>> introduce a significant amount of latency in some cases.
> > > >> Plus,
> > > >>>>>>>>> it's a
> > > >>>>>>>>>>> more
> > > >>>>>>>>>>>>> direct measurement, if you really wanted to know (for the
> > > >>>>>>> purposes
> > > >>>>>>>>> of
> > > >>>>>>>>>>> IQ
> > > >>>>>>>>>>>>> or something) how long it takes source events to "show
> up" at
> > > >>>>>>> the
> > > >>>>>>>>>>> store.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I think actually recording it at every processor could be
> > > >>>>>>>>> expensive,
> > > >>>>>>>>>>> but we
> > > >>>>>>>>>>>>> already record a bunch of metrics at the node level.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> B1:
> > > >>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if
> it's
> > > >> a
> > > >>>>>>> poor
> > > >>>>>>>>>>> metric
> > > >>>>>>>>>>>>> for operational purposes, a lot of people might expect
> to see
> > > >>>>>>>>> "mean".
> > > >>>>>>>>>>>>> Actually,
> > > >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to
> leave
> > > >> it
> > > >>>>>>>>> off?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I might suggest:
> > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> B3:
> > > >>>>>>>>>>>>> I agree we should include late records (though not the
> ones
> > > >> we
> > > >>>>>>>>> drop).
> > > >>>>>>>>>>>>> It may be spiky, but only when there are legitimately
> some
> > > >>>>>>> records
> > > >>>>>>>>>>> with a
> > > >>>>>>>>>>>>> high end-to-end latency, which is the whole point of
> these
> > > >>>>>>> metrics.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> That's it! I don't think I have any other feedback, other
> > > >> than
> > > >>>>>>> a
> > > >>>>>>>>>>> request to
> > > >>>>>>>>>>>>> also report "min".
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> -John
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
> > > >>>>>>> latency
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>> task
> > > >>>>>>>>>>>>>> and the latency spent sitting in intermediate topics. I
> have
> > > >>>>>>> a
> > > >>>>>>>>>>> similar
> > > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency
> metric of
> > > >>>>>>> a
> > > >>>>>>>>> task A
> > > >>>>>>>>>>>>>> actually measures the latency of the sub-topology up-to
> but
> > > >>>>>>> not
> > > >>>>>>>>>>> including
> > > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency"
> itself,
> > > >>>>>>> since
> > > >>>>>>>>>>> today we
> > > >>>>>>>>>>>>>> already have several "latency" metrics already which are
> > > >>>>>>>>> measuring
> > > >>>>>>>>>>>>> elapsed
> > > >>>>>>>>>>>>>> system-time for processing a record / etc, while here
> we are
> > > >>>>>>>>>>> comparing
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> > > >>>>>>> (note we
> > > >>>>>>>>>>> already
> > > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case
> recording at
> > > >>>>>>> the
> > > >>>>>>>>>>>>>> system-time before we start processing the record sounds
> > > >> more
> > > >>>>>>>>>>> natural.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > >>>>>>> processor-node-level
> > > >>>>>>>>> DEBUG
> > > >>>>>>>>>>>>>> metric is worth to add, given that we already have a
> > > >>>>>>> task-level
> > > >>>>>>>>>>>>> processing
> > > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e
> latency is
> > > >>>>>>>>> similar
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> task-level e2e latency + task-level processing latency.
> > > >>>>>>>>> Personally I
> > > >>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>> having a task-level record-staleness metric is
> sufficient.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Guozhang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > >>>>>>>>>>>>> sophie@confluent.io>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge
> for
> > > >>>>>>> this
> > > >>>>>>>>>>>>> specific
> > > >>>>>>>>>>>>>>> metric, as
> > > >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper
> *bound on
> > > >>>>>>> the
> > > >>>>>>>>>>> latency
> > > >>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that
> would
> > > >>>>>>>>> hopefully
> > > >>>>>>>>>>>>> give a
> > > >>>>>>>>>>>>>>> good
> > > >>>>>>>>>>>>>>> sense of what *most* records will experience, and what
> > > >>>>>>> *close
> > > >>>>>>>>> to
> > > >>>>>>>>>>> all*
> > > >>>>>>>>>>>>>>> records
> > > >>>>>>>>>>>>>>> will.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> However I'm not married to these specific numbers and
> > > >>>>>>> could be
> > > >>>>>>>>>>>>> convinced.
> > > >>>>>>>>>>>>>>> Would be especially interested in hearing from users on
> > > >>>>>>> this.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop
> latency" in
> > > >>>>>>>>> this KIP
> > > >>>>>>>>>>>>> since
> > > >>>>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>>> can always compute it themselves by subtracting the
> > > >>>>>>> previous
> > > >>>>>>>>> node's
> > > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way
> since
> > > >>>>>>>>> you can
> > > >>>>>>>>>>>>> always
> > > >>>>>>>>>>>>>>> compute one from the other, but I think the end-to-end
> > > >>>>>>> latency
> > > >>>>>>>>>>> feels
> > > >>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> > > >>>>>>> bottlenecks
> > > >>>>>>>>> in the
> > > >>>>>>>>>>>>>>> topology but
> > > >>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to
> be
> > > >>>>>>>>> reflected
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> certain parts
> > > >>>>>>>>>>>>>>> of the topology. For example this might be useful for
> users
> > > >>>>>>>>> who are
> > > >>>>>>>>>>>>>>> wondering
> > > >>>>>>>>>>>>>>> roughly when a record that was just produced will be
> > > >>>>>>> included
> > > >>>>>>>>> in
> > > >>>>>>>>>>> their
> > > >>>>>>>>>>>>> IQ
> > > >>>>>>>>>>>>>>> results.
> > > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I
> didn't
> > > >>>>>>> make
> > > >>>>>>>>>>> that
> > > >>>>>>>>>>>>> clear
> > > >>>>>>>>>>>>>>> enough
> > > >>>>>>>>>>>>>>> in the KIP's motivation.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> > > >>>>>>> short
> > > >>>>>>>>>>> answer is
> > > >>>>>>>>>>>>>>> "yes",
> > > >>>>>>>>>>>>>>> we will include late records. I added a paragraph to
> the
> > > >>>>>>> end
> > > >>>>>>>>> of the
> > > >>>>>>>>>>>>>>> Proposed
> > > >>>>>>>>>>>>>>> Changes section explaining the reasoning here, please
> let
> > > >>>>>>> me
> > > >>>>>>>>> know
> > > >>>>>>>>>>> if
> > > >>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>> any concerns.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > > >>>>>>>>>>> "process-latency",
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> metric
> > > >>>>>>>>>>>>>>> reflects the time for the literal Node#process method
> to
> > > >>>>>>> run
> > > >>>>>>>>>>> whereas
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> metric
> > > >>>>>>>>>>>>>>> would always be measured relative to the event
> timestamp.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> That said, the naming collision there is pretty
> confusing
> > > >>>>>>> so
> > > >>>>>>>>> I've
> > > >>>>>>>>>>>>> renamed
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I
> feel
> > > >>>>>>> better
> > > >>>>>>>>>>>>> reflects
> > > >>>>>>>>>>>>>>> the nature
> > > >>>>>>>>>>>>>>> of the metric anyway.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for the feedback!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> > > >>>>>>>>> important
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99
> and
> > > >>>>>>> max
> > > >>>>>>>>>>>>> percentiles?
> > > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> > > >>>>>>> well
> > > >>>>>>>>> in
> > > >>>>>>>>>>>>> production
> > > >>>>>>>>>>>>>>>> systems.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative,
> I.E
> > > >>>>>>> if a
> > > >>>>>>>>>>> record
> > > >>>>>>>>>>>>> goes
> > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > > >>>>>>> T(B->C) +
> > > >>>>>>>>>>> T(A->B) +
> > > >>>>>>>>>>>>>>> T(A)
> > > >>>>>>>>>>>>>>>> and so on, where P() represents the captured latency,
> > > >>>>>>> and T()
> > > >>>>>>>>>>>>> represents
> > > >>>>>>>>>>>>>>>> the time for transiting the records between two nodes,
> > > >>>>>>>>> including
> > > >>>>>>>>>>>>>>> processing
> > > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> > > >>>>>>>>> T(A->B)
> > > >>>>>>>>>>> are
> > > >>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> > > >>>>>>> there
> > > >>>>>>>>> is a
> > > >>>>>>>>>>>>> spike in
> > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> > > >>>>>>> time.
> > > >>>>>>>>> In the
> > > >>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> > > >>>>>>> record
> > > >>>>>>>>> exits
> > > >>>>>>>>>>>>> from the
> > > >>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
> > > >>>>>>> inside
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> funnel. Do
> > > >>>>>>>>>>>>>>>> you think we could have separate treatment for sink
> > > >>>>>>> nodes and
> > > >>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time
> receiving
> > > >>>>>>> the
> > > >>>>>>>>>>> record
> > > >>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just
> want to
> > > >>>>>>>>> discuss
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> > > >>>>>>> well,
> > > >>>>>>>>> they
> > > >>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>> create some really spiky graphs when the out-of-order
> > > >>>>>>>>> records are
> > > >>>>>>>>>>>>>>>> interleaving with on time records. Should we also
> supply
> > > >>>>>>> a
> > > >>>>>>>>> smooth
> > > >>>>>>>>>>>>> version
> > > >>>>>>>>>>>>>>>> of the latency metrics, or user should just take care
> of
> > > >>>>>>> it
> > > >>>>>>>>> by
> > > >>>>>>>>>>>>> themself?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed
> its
> > > >>>>>>>>> relation
> > > >>>>>>>>>>>>> with our
> > > >>>>>>>>>>>>>>>> existing processing latency metrics, could you add
> some
> > > >>>>>>>>> context
> > > >>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial
> for
> > > >>>>>>> the
> > > >>>>>>>>>>> best?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Boyang
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > >>>>>>>>>>>>> sophie@confluent.io
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hey all,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> > > >>>>>>> to
> > > >>>>>>>>> add
> > > >>>>>>>>>>>>> end-to-end
> > > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>> Sophie
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>> -- Guozhang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
Thanks, Sophie!

I think this makes perfect sense. It will be much more intuitive to use the metric for the stated motivation this way.  I’d be in favor of the proposal after this update. 

Thanks again for taking this on,
-John

On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman wrote:
> Hi all,
> 
> I'd like to clarify/modify one aspect of this KIP, which is to record the
> staleness
> at the *completion* of the record's processing by the operator/task in
> question,
> rather than on the intake. The task-level metrics will be recorded at the
> sink
> node instead of at the source, and the operator-level metrics will be
> recorded
> at the end of the operation.
> 
> The stated purpose and intended usefulness of this KIP is to give users a
> way
> to gauge roughly how long it takes for a record to be reflected in the
> "results",
> whether these results are being read from an output topic or through IQ. To
> take the IQ example, the results of a record are obviously not visible until
> *after* that node has finished processing it. The staleness, while still
> potentially
> useful as it can impact the *way* a record is processed in a stateful and
> time-
> dependent operator, is not part of the problem this KIP specifically set out
> to solve.
> 
> In light of this I think it's appropriate to revert the name change back to
> include
> latency, since "staleness" as described above only makes sense when
> measuring
> relative to the arrival of a record at a task/node. I'd propose to save the
> term
> "staleness" for that particular meaning, and adopt Matthias's suggestion of
> "record-e2e-latency" for this.
> 
> Thanks for hanging in there all. Please let me know if you have any
> concerns about
> this change!
> 
> Sophie
> 
> On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org> wrote:
> 
> > I am also happy with max/min/99/90. And I buy your naming argument about
> > staleness vs latency.
> >
> >
> > -Matthias
> >
> > On 5/15/20 12:24 PM, Boyang Chen wrote:
> > > Hey Sophie,
> > >
> > > 90/99/min/max make sense to me.
> > >
> > > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> > sophie@confluent.io>
> > > wrote:
> > >
> > >> @Matthias
> > >> Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark
> > of
> > >> the video
> > >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> > >>
> > >> And that's what he says about the 95th percentile! Imagine what he would
> > >> say about
> > >> the 50th :P
> > >>
> > >> But seriously, since we can't seem to agree that the mean or 50th
> > >> percentile is actually
> > >> useful I'm inclined to resurrect my original proposal, neither. But I
> > think
> > >> that's a good
> > >> argument against the 75th, which I admittedly chose somewhat
> > arbitrarily as
> > >> an
> > >> intermediate between the 50th and the higher percentiles. How about:
> > >>
> > >> -max
> > >> -p99
> > >> -p90
> > >> -min
> > >>
> > >> with p50/mean still up for debate if anyone feels strongly for either of
> > >> them.
> > >>
> > >> Regarding the name, I've already flip-flopped on this so I'm definitely
> > >> still open to
> > >> further arguments. But the reason for changing it from
> > end-to-end-latency
> > >> (which
> > >> is similar to what you propose) is that this metric technically reflects
> > >> how old (ie how "stale")
> > >> the record is when it's *received* by the operator, not when it's
> > processed
> > >> by the operator.
> > >> It seemed like there was the potential for confusion that
> > >> "end-to-end-latency" might
> > >> represent the latency from the event creation to the time the processor
> > is
> > >> done
> > >> processing it.
> > >>
> > >> @John
> > >> I'd rather err on the side of "not-enough" metrics as we can always add
> > >> this to the
> > >> stateless metrics later on. If we decide to measure the time at every
> > node
> > >> and don't
> > >> find any evidence of a serious performance impact, and users indicate
> > they
> > >> would
> > >> like to see this metric at all nodes, then we can easily start reporting
> > >> them as well.
> > >> WDYT?
> > >>
> > >> That said, sink nodes seem like a reasonable exception to the rule.
> > >> Obviously users
> > >> should be able to detect the time when the record reaches the output
> > topic
> > >> but that
> > >> still leaves a gap in understanding how long the production latency was.
> > >> This mirrors
> > >> the consumption latency that is exposed by the task-level metrics, which
> > >> are measured
> > >> at the source node. For good symmetry what if we actually expose both
> > the
> > >> source
> > >> and sink latency at the task-level? ie report both sets of statistical
> > >> measurements with
> > >> the additional tag -source/-sink
> > >>
> > >> @Bill
> > >> Thanks for the comment about regarding the min! I hadn't considered that
> > >> and it's
> > >> quite useful to think about how and what is useful from a users point of
> > >> view.
> > >>
> > >> Regarding your second. point, I'm inclined to leave that as an
> > >> implementation detail
> > >> but my take would be that the user should be allowed to control the
> > record
> > >> timestamp
> > >> used for this with the timestamp extractor. My impression is that users
> > may
> > >> often embed
> > >> the actual event time in the payload for whatever reason, and this
> > >> represents the "true"
> > >> timestamp as far as the Streams topology is concerned.
> > >>
> > >>
> > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com> wrote:
> > >>
> > >>> Thanks for the KIP, Sophie, this will be a useful metric to add.
> > >>>
> > >>> Regarding tracking min, I  think it could be valuable for users to
> > >> discern
> > >>> which part of their topologies are more efficient since this is a
> > >>> task-level metric.  I realize everyone seems to be on board with
> > >> including
> > >>> min anyway, but I wanted to add my 2 cents on this topic should we
> > decide
> > >>> to revisit adding min or not.
> > >>>
> > >>> I do have a question regarding the calculation of staleness.
> > >>> Is there going to be a consideration for timestamp extractors? Users
> > >> could
> > >>> prefer to use a timestamp embedded in the payload, and it could skew
> > the
> > >>> measurements.
> > >>> I was wondering if we should specify in the KIP if setting the arrival
> > >> time
> > >>> is always going to come from the record timestamp, or is this an
> > >>> implementation detail we can cover in the PR?
> > >>>
> > >>> Thanks!
> > >>> Bill
> > >>>
> > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
> > >> wrote:
> > >>>
> > >>>> Thanks for the KIP Sophie.
> > >>>>
> > >>>> I think it's not useful to record the avg/mean; it sensitive to
> > >>>> outliers. We should rather track the median (50th percentile).
> > >>>>
> > >>>> Not sure if tracking min is useful, but I am also ok to track it.
> > >>>>
> > >>>> However, I find it odd to track 75th percentile. Standard measures
> > >> would
> > >>>> the 90th or 95th -- I guess we don't need both, so maybe picking 90th
> > >>>> might be more useful?
> > >>>>
> > >>>> About the name: "staleness" wound really odd, and if fact the metric
> > >>>> does capture "latency" so we should call it "latency". I understand
> > the
> > >>>> issue that we already have a latency metric. So maybe we could call it
> > >>>> `record-e2e-latency-*` ?
> > >>>>
> > >>>> While I agree that we should include out-or-order data (the KIP should
> > >>>> talk about `out-of-order` data, not `late` data; data is only `late`
> > if
> > >>>> it's out-of-order and if it's dropped), I don't really understand why
> > >>>> the new metric would help to configure grace period or retention time?
> > >>>> As you mention in the KIP, both are define as max difference of
> > >>>> `event-time - stream-time` and thus the new metric that takes
> > >>>> system-/wallclock-time into account does not seem to help at all.
> > >>>>
> > >>>>
> > >>>> Btw: there is a great talk about "How NOT to Measure Latency" by Gil
> > >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > >>>>
> > >>>>
> > >>>> -Matthias
> > >>>>
> > >>>>
> > >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> > >>>>> Hi Sophie,
> > >>>>>
> > >>>>> It seems like there would still be plenty of use cases for recording
> > >>>>> this metric at all processors and not just stateful ones, but I'm
> > >> happy
> > >>>>> to suspend my arguments for now. Since you're proposing to keep
> > >>>>> them at the processor-node level, it will be seamless later to add
> > >>>>> in the stateless processors if we want. As a wise man once said,
> > >>>>> "Adding is always easier than removing."
> > >>>>>
> > >>>>> Regarding the time measurement, it's an implementation detail
> > >>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
> > >>>>> prefer to measure the system time again when recording the
> > >>>>> metric. I don't think we've seen any evidence that proves this
> > >>>>> would harm performance, and the amount of inaccuracy using
> > >>>>> the cached system time could incur is actually substantial. But,
> > >>>>> if you want to just "not mention this" in the KIP, we can defer to
> > >>>>> the actual PR discussion, at which time we're in a better position
> > >>>>> to use benchmarks, etc., to make the call.
> > >>>>>
> > >>>>> Along the lines of the measurement accuracy discussion, one
> > >>>>> minor thought I had is that maybe we should consider measuring
> > >>>>> the task staleness metric at the sink, rather than the source, so
> > >> that
> > >>>>> it includes the processing latency of the task itself, not just the
> > >>>> latency
> > >>>>> of everything up to, but not including, the task (which seems
> > >> confusing
> > >>>>> for users). I guess this could also be an implementation detail,
> > >>> though.
> > >>>>>
> > >>>>> Thanks for the update,
> > >>>>> -John
> > >>>>>
> > >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > >>>>>> Hey all,
> > >>>>>>
> > >>>>>> After discussing with Bruno I'd like to propose a small amendment,
> > >>>>>> which is to record the processor-node-level metrics only for
> > >>> *stateful*
> > >>>>>> *operators*. They would still be considered a "processor-node-level"
> > >>>>>> metric and not a "state-store-level" metric as the staleness is
> > >> still
> > >>>>>> a property of the node rather than of the state itself. However, it
> > >>>> seems
> > >>>>>> that this information is primarily useful for stateful operators
> > >> that
> > >>>> might
> > >>>>>> be exposing state via IQ or otherwise dependent on the record time
> > >>>>>> unlike a stateless operator.
> > >>>>>>
> > >>>>>> It's worth calling out that recent performance improvements to the
> > >>>> metrics
> > >>>>>> framework mean that we no longer fetch the system time at the
> > >> operator
> > >>>>>> level, but only once per task. In other words the system time is not
> > >>>> updated
> > >>>>>> between each process as a record flows through the subtopology, so
> > >>>>>> debugging the processor-level latency via the stateleness will not
> > >> be
> > >>>>>> possible.Note that this doesn't mean the operator-level metrics are
> > >>> not
> > >>>>>> *useful* relative to the task-level metric. Upstream caching and/or
> > >>>>>> suppression
> > >>>>>> can still cause a record's staleness at some downstream stateful
> > >>>> operator
> > >>>>>> to deviate from the task-level staleness (recorded at the source
> > >>> node).
> > >>>>>>
> > >>>>>> Please let me know if you have any concerns about this change. The
> > >>>>>> KIP has been updated with the new proposal
> > >>>>>>
> > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Sophie,
> > >>>>>>>
> > >>>>>>> Thank you for the KIP.
> > >>>>>>>
> > >>>>>>> The KIP looks good to me.
> > >>>>>>>
> > >>>>>>> 50th percentile:
> > >>>>>>> I think we do not need it now. If we need it, we can add it. Here
> > >> the
> > >>>>>>> old truism applies: Adding is always easier than removing.
> > >>>>>>>
> > >>>>>>> processor-node-level metrics:
> > >>>>>>> I think it is good to have the staleness metrics also on
> > >>>>>>> processor-node-level. If we do not want to record them on all
> > >>>>>>> processor nodes, you could restrict the recording to stateful
> > >>>>>>> processor-nodes, since those are the ones that would benefit most
> > >>> from
> > >>>>>>> the staleness metrics.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Bruno
> > >>>>>>>
> > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > >>>> sophie@confluent.io>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Yeah, the specific reason was just to align with the current
> > >>> metrics.
> > >>>>>>>>
> > >>>>>>>> Is it better to conform than to be right? History has a lot to say
> > >>> on
> > >>>>>>> that
> > >>>>>>>> matter
> > >>>>>>>> but I'm not sure how much of it applies to the fine details of
> > >>> metrics
> > >>>>>>>> naming :P
> > >>>>>>>>
> > >>>>>>>> More seriously, I figured if people are looking at this metric
> > >>> they're
> > >>>>>>>> likely to
> > >>>>>>>> be looking at all the others. Then naming this one "-mean" would
> > >>>> probably
> > >>>>>>>> lead some to conclude that the "-avg" suffix in the other metrics
> > >>> has
> > >>>> a
> > >>>>>>>> different meaning.
> > >>>>>>>>
> > >>>>>>>> As for the percentiles, I actually like p99 (and p75) better. I'll
> > >>>> swap
> > >>>>>>>> that out
> > >>>>>>>>
> > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vvcephei@apache.org
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks Sophie,
> > >>>>>>>>>
> > >>>>>>>>> I hope this isn't too nit-picky, but is there a reason to choose
> > >>>> "avg"
> > >>>>>>>>> instead
> > >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> > >> oversensitive
> > >>>>>>> because
> > >>>>>>>>> of the mistake I just made earlier, but it strikes me that "avg"
> > >> is
> > >>>>>>>>> actually
> > >>>>>>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
> > >>> is
> > >>>>>>>>> specific.
> > >>>>>>>>> I see other Kafka metrics with "avg", but none with "mean"; was
> > >>> that
> > >>>>>>> the
> > >>>>>>>>> reason? If so, I'm +1.
> > >>>>>>>>>
> > >>>>>>>>> Regarding the names of the percentile, I actually couldn't find
> > >>> _any_
> > >>>>>>> other
> > >>>>>>>>> metrics that use percentile. Was there a reason to choose "99th"
> > >> as
> > >>>>>>> opposed
> > >>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm just
> > >>>>>>> primarily
> > >>>>>>>>> asking
> > >>>>>>>>> for consistency's sake.
> > >>>>>>>>>
> > >>>>>>>>> Thanks again,
> > >>>>>>>>> -John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > >>>>>>>>>> Alright, I can get behind adding the min metric for the sake of
> > >>>>>>> pretty
> > >>>>>>>>>> graphs
> > >>>>>>>>>> (and trivial computation).
> > >>>>>>>>>>
> > >>>>>>>>>> I'm still on the fence regarding the mean (or 50th percentile)
> > >>> but I
> > >>>>>>> can
> > >>>>>>>>> see
> > >>>>>>>>>> how users might expect it and find it a bit disorienting not to
> > >>>>>>> have. So
> > >>>>>>>>> the
> > >>>>>>>>>> updated proposed metrics are
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>    - record-staleness-max [ms]
> > >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> > >>>>>>>>>>    - record-staleness-min [ms]
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> > >> vvcephei@apache.org
> > >>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
> > >> guess
> > >>>>>>> the
> > >>>>>>>>> mean
> > >>>>>>>>>>> seems more interesting to me than the median, but neither are
> > >> as
> > >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Min isn’t really important for any SLAs, but it does round out
> > >>> the
> > >>>>>>>>> mental
> > >>>>>>>>>>> picture of the distribution. I’ve always graphed min along with
> > >>> the
> > >>>>>>>>> other
> > >>>>>>>>>>> metrics to help me understand how fast the system can be, which
> > >>>>>>> helps
> > >>>>>>>>> in
> > >>>>>>>>>>> optimization decisions. It’s also a relatively inexpensive
> > >> metric
> > >>>>>>> to
> > >>>>>>>>>>> compute, so it might be nice to just throw it in.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > >>>>>>>>>>>> G1:
> > >>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to the
> > >>>>>>> specific
> > >>>>>>>>>>> task"
> > >>>>>>>>>>>> but
> > >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the point
> > >> home
> > >>>>>>>>> better.
> > >>>>>>>>>>> So
> > >>>>>>>>>>>> it's the
> > >>>>>>>>>>>> "staleness of the record when it is received by that task" --
> > >>>>>>> will
> > >>>>>>>>> update
> > >>>>>>>>>>>> the KIP
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> B1/J:
> > >>>>>>>>>>>> I'm struggling to imagine a case where the min would actually
> > >> be
> > >>>>>>>>> useful,
> > >>>>>>>>>>>> rather than
> > >>>>>>>>>>>> just intellectually interesting. I don't feel strongly that we
> > >>>>>>>>> shouldn't
> > >>>>>>>>>>>> add it, but that's
> > >>>>>>>>>>>> why I didn't include it from the start. Can you enlighten me
> > >>>>>>> with an
> > >>>>>>>>>>>> example?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I was also vaguely concerned about the overhead of adding
> > >>>>>>> multiple
> > >>>>>>>>>>>> percentile
> > >>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
> > >>> performance
> > >>>>>>>>> hit we
> > >>>>>>>>>>>> take on
> > >>>>>>>>>>>> metrics computation?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile would be
> > >>> the
> > >>>>>>>>> median
> > >>>>>>>>>>>> not the
> > >>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > >>>>>>> percentile, or
> > >>>>>>>>>>> just
> > >>>>>>>>>>>> one
> > >>>>>>>>>>>> of the two?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks all!
> > >>>>>>>>>>>> Sophie
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > >>>>>>> vvcephei@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Just some comments on the discussion so far:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> B2/G1:
> > >>>>>>>>>>>>> In principle, it shouldn't matter whether we report "spans"
> > >> or
> > >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
> > >>>>>>> pretty
> > >>>>>>>>>>>>> difficult to really measure (like time spent waiting in the
> > >>>>>>>>> topics, or
> > >>>>>>>>>>>>> time from the event happening to the ETL producer choosing to
> > >>>>>>> send
> > >>>>>>>>> it,
> > >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In other words, it's practically easier to compute spans by
> > >>>>>>>>> subtracting
> > >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
> > >>>>>>> spans.
> > >>>>>>>>> You
> > >>>>>>>>>>>>> can even just consider that the span computation from e2e
> > >>>>>>> always
> > >>>>>>>>> just
> > >>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
> > >> latency
> > >>>>>>>>> from
> > >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the end
> > >> you
> > >>>>>>>>> care
> > >>>>>>>>>>> about.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> It seems like people really prefer to have spans when they
> > >> are
> > >>>>>>>>>>> debugging
> > >>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
> > >>>>>>> measurement
> > >>>>>>>>>>>>> that basically every person/application cares about and
> > >> should
> > >>>>>>> be
> > >>>>>>>>>>>>> monitoring.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Altogether, it really seem to provide more value to more
> > >>>>>>> people if
> > >>>>>>>>> we
> > >>>>>>>>>>>>> report
> > >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
> > >> think
> > >>>>>>> I
> > >>>>>>>>> have
> > >>>>>>>>>>> no
> > >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> G2:
> > >>>>>>>>>>>>> I think the processor-node metric is nice, since the inside
> > >> of
> > >>>>>>> a
> > >>>>>>>>> task
> > >>>>>>>>>>> can
> > >>>>>>>>>>>>> introduce a significant amount of latency in some cases.
> > >> Plus,
> > >>>>>>>>> it's a
> > >>>>>>>>>>> more
> > >>>>>>>>>>>>> direct measurement, if you really wanted to know (for the
> > >>>>>>> purposes
> > >>>>>>>>> of
> > >>>>>>>>>>> IQ
> > >>>>>>>>>>>>> or something) how long it takes source events to "show up" at
> > >>>>>>> the
> > >>>>>>>>>>> store.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I think actually recording it at every processor could be
> > >>>>>>>>> expensive,
> > >>>>>>>>>>> but we
> > >>>>>>>>>>>>> already record a bunch of metrics at the node level.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> B1:
> > >>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if it's
> > >> a
> > >>>>>>> poor
> > >>>>>>>>>>> metric
> > >>>>>>>>>>>>> for operational purposes, a lot of people might expect to see
> > >>>>>>>>> "mean".
> > >>>>>>>>>>>>> Actually,
> > >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to leave
> > >> it
> > >>>>>>>>> off?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I might suggest:
> > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> B3:
> > >>>>>>>>>>>>> I agree we should include late records (though not the ones
> > >> we
> > >>>>>>>>> drop).
> > >>>>>>>>>>>>> It may be spiky, but only when there are legitimately some
> > >>>>>>> records
> > >>>>>>>>>>> with a
> > >>>>>>>>>>>>> high end-to-end latency, which is the whole point of these
> > >>>>>>> metrics.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> That's it! I don't think I have any other feedback, other
> > >> than
> > >>>>>>> a
> > >>>>>>>>>>> request to
> > >>>>>>>>>>>>> also report "min".
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> -John
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
> > >>>>>>> latency
> > >>>>>>>>> of the
> > >>>>>>>>>>>>> task
> > >>>>>>>>>>>>>> and the latency spent sitting in intermediate topics. I have
> > >>>>>>> a
> > >>>>>>>>>>> similar
> > >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
> > >>>>>>> a
> > >>>>>>>>> task A
> > >>>>>>>>>>>>>> actually measures the latency of the sub-topology up-to but
> > >>>>>>> not
> > >>>>>>>>>>> including
> > >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
> > >>>>>>> since
> > >>>>>>>>>>> today we
> > >>>>>>>>>>>>>> already have several "latency" metrics already which are
> > >>>>>>>>> measuring
> > >>>>>>>>>>>>> elapsed
> > >>>>>>>>>>>>>> system-time for processing a record / etc, while here we are
> > >>>>>>>>>>> comparing
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> > >>>>>>> (note we
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>>> have a "record-lateness" metric), in which case recording at
> > >>>>>>> the
> > >>>>>>>>>>>>>> system-time before we start processing the record sounds
> > >> more
> > >>>>>>>>>>> natural.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> > >>>>>>> processor-node-level
> > >>>>>>>>> DEBUG
> > >>>>>>>>>>>>>> metric is worth to add, given that we already have a
> > >>>>>>> task-level
> > >>>>>>>>>>>>> processing
> > >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
> > >>>>>>>>> similar
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> task-level e2e latency + task-level processing latency.
> > >>>>>>>>> Personally I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>>> having a task-level record-staleness metric is sufficient.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Guozhang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > >>>>>>>>>>>>> sophie@confluent.io>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
> > >>>>>>> this
> > >>>>>>>>>>>>> specific
> > >>>>>>>>>>>>>>> metric, as
> > >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
> > >>>>>>> the
> > >>>>>>>>>>> latency
> > >>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that would
> > >>>>>>>>> hopefully
> > >>>>>>>>>>>>> give a
> > >>>>>>>>>>>>>>> good
> > >>>>>>>>>>>>>>> sense of what *most* records will experience, and what
> > >>>>>>> *close
> > >>>>>>>>> to
> > >>>>>>>>>>> all*
> > >>>>>>>>>>>>>>> records
> > >>>>>>>>>>>>>>> will.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> However I'm not married to these specific numbers and
> > >>>>>>> could be
> > >>>>>>>>>>>>> convinced.
> > >>>>>>>>>>>>>>> Would be especially interested in hearing from users on
> > >>>>>>> this.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
> > >>>>>>>>> this KIP
> > >>>>>>>>>>>>> since
> > >>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>> can always compute it themselves by subtracting the
> > >>>>>>> previous
> > >>>>>>>>> node's
> > >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way since
> > >>>>>>>>> you can
> > >>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>> compute one from the other, but I think the end-to-end
> > >>>>>>> latency
> > >>>>>>>>>>> feels
> > >>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> > >>>>>>> bottlenecks
> > >>>>>>>>> in the
> > >>>>>>>>>>>>>>> topology but
> > >>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to be
> > >>>>>>>>> reflected
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>>> certain parts
> > >>>>>>>>>>>>>>> of the topology. For example this might be useful for users
> > >>>>>>>>> who are
> > >>>>>>>>>>>>>>> wondering
> > >>>>>>>>>>>>>>> roughly when a record that was just produced will be
> > >>>>>>> included
> > >>>>>>>>> in
> > >>>>>>>>>>> their
> > >>>>>>>>>>>>> IQ
> > >>>>>>>>>>>>>>> results.
> > >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
> > >>>>>>> make
> > >>>>>>>>>>> that
> > >>>>>>>>>>>>> clear
> > >>>>>>>>>>>>>>> enough
> > >>>>>>>>>>>>>>> in the KIP's motivation.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> > >>>>>>> short
> > >>>>>>>>>>> answer is
> > >>>>>>>>>>>>>>> "yes",
> > >>>>>>>>>>>>>>> we will include late records. I added a paragraph to the
> > >>>>>>> end
> > >>>>>>>>> of the
> > >>>>>>>>>>>>>>> Proposed
> > >>>>>>>>>>>>>>> Changes section explaining the reasoning here, please let
> > >>>>>>> me
> > >>>>>>>>> know
> > >>>>>>>>>>> if
> > >>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>> any concerns.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > >>>>>>>>>>> "process-latency",
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> metric
> > >>>>>>>>>>>>>>> reflects the time for the literal Node#process method to
> > >>>>>>> run
> > >>>>>>>>>>> whereas
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> metric
> > >>>>>>>>>>>>>>> would always be measured relative to the event timestamp.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> That said, the naming collision there is pretty confusing
> > >>>>>>> so
> > >>>>>>>>> I've
> > >>>>>>>>>>>>> renamed
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
> > >>>>>>> better
> > >>>>>>>>>>>>> reflects
> > >>>>>>>>>>>>>>> the nature
> > >>>>>>>>>>>>>>> of the metric anyway.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for the feedback!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > >>>>>>>>>>>>> reluctanthero104@gmail.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> > >>>>>>>>> important
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> A couple of questions and ideas:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
> > >>>>>>> max
> > >>>>>>>>>>>>> percentiles?
> > >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> > >>>>>>> well
> > >>>>>>>>> in
> > >>>>>>>>>>>>> production
> > >>>>>>>>>>>>>>>> systems.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
> > >>>>>>> if a
> > >>>>>>>>>>> record
> > >>>>>>>>>>>>> goes
> > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > >>>>>>> T(B->C) +
> > >>>>>>>>>>> T(A->B) +
> > >>>>>>>>>>>>>>> T(A)
> > >>>>>>>>>>>>>>>> and so on, where P() represents the captured latency,
> > >>>>>>> and T()
> > >>>>>>>>>>>>> represents
> > >>>>>>>>>>>>>>>> the time for transiting the records between two nodes,
> > >>>>>>>>> including
> > >>>>>>>>>>>>>>> processing
> > >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> > >>>>>>>>> T(A->B)
> > >>>>>>>>>>> are
> > >>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> > >>>>>>> there
> > >>>>>>>>> is a
> > >>>>>>>>>>>>> spike in
> > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> > >>>>>>> time.
> > >>>>>>>>> In the
> > >>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> > >>>>>>> record
> > >>>>>>>>> exits
> > >>>>>>>>>>>>> from the
> > >>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
> > >>>>>>> inside
> > >>>>>>>>> the
> > >>>>>>>>>>>>> funnel. Do
> > >>>>>>>>>>>>>>>> you think we could have separate treatment for sink
> > >>>>>>> nodes and
> > >>>>>>>>>>> other
> > >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time receiving
> > >>>>>>> the
> > >>>>>>>>>>> record
> > >>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
> > >>>>>>>>> discuss
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> > >>>>>>> well,
> > >>>>>>>>> they
> > >>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>> create some really spiky graphs when the out-of-order
> > >>>>>>>>> records are
> > >>>>>>>>>>>>>>>> interleaving with on time records. Should we also supply
> > >>>>>>> a
> > >>>>>>>>> smooth
> > >>>>>>>>>>>>> version
> > >>>>>>>>>>>>>>>> of the latency metrics, or user should just take care of
> > >>>>>>> it
> > >>>>>>>>> by
> > >>>>>>>>>>>>> themself?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
> > >>>>>>>>> relation
> > >>>>>>>>>>>>> with our
> > >>>>>>>>>>>>>>>> existing processing latency metrics, could you add some
> > >>>>>>>>> context
> > >>>>>>>>>>> on
> > >>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
> > >>>>>>> the
> > >>>>>>>>>>> best?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Boyang
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > >>>>>>>>>>>>> sophie@confluent.io
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> > >>>>>>> to
> > >>>>>>>>> add
> > >>>>>>>>>>>>> end-to-end
> > >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>> Sophie
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>> -- Guozhang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Hi all,

I'd like to clarify/modify one aspect of this KIP, which is to record the
staleness
at the *completion* of the record's processing by the operator/task in
question,
rather than on the intake. The task-level metrics will be recorded at the
sink
node instead of at the source, and the operator-level metrics will be
recorded
at the end of the operation.

The stated purpose and intended usefulness of this KIP is to give users a
way
to gauge roughly how long it takes for a record to be reflected in the
"results",
whether these results are being read from an output topic or through IQ. To
take the IQ example, the results of a record are obviously not visible until
*after* that node has finished processing it. The staleness, while still
potentially
useful as it can impact the *way* a record is processed in a stateful and
time-
dependent operator, is not part of the problem this KIP specifically set out
to solve.

In light of this I think it's appropriate to revert the name change back to
include
latency, since "staleness" as described above only makes sense when
measuring
relative to the arrival of a record at a task/node. I'd propose to save the
term
"staleness" for that particular meaning, and adopt Matthias's suggestion of
"record-e2e-latency" for this.

Thanks for hanging in there all. Please let me know if you have any
concerns about
this change!

Sophie

On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax <mj...@apache.org> wrote:

> I am also happy with max/min/99/90. And I buy your naming argument about
> staleness vs latency.
>
>
> -Matthias
>
> On 5/15/20 12:24 PM, Boyang Chen wrote:
> > Hey Sophie,
> >
> > 90/99/min/max make sense to me.
> >
> > On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <
> sophie@confluent.io>
> > wrote:
> >
> >> @Matthias
> >> Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark
> of
> >> the video
> >> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
> >>
> >> And that's what he says about the 95th percentile! Imagine what he would
> >> say about
> >> the 50th :P
> >>
> >> But seriously, since we can't seem to agree that the mean or 50th
> >> percentile is actually
> >> useful I'm inclined to resurrect my original proposal, neither. But I
> think
> >> that's a good
> >> argument against the 75th, which I admittedly chose somewhat
> arbitrarily as
> >> an
> >> intermediate between the 50th and the higher percentiles. How about:
> >>
> >> -max
> >> -p99
> >> -p90
> >> -min
> >>
> >> with p50/mean still up for debate if anyone feels strongly for either of
> >> them.
> >>
> >> Regarding the name, I've already flip-flopped on this so I'm definitely
> >> still open to
> >> further arguments. But the reason for changing it from
> end-to-end-latency
> >> (which
> >> is similar to what you propose) is that this metric technically reflects
> >> how old (ie how "stale")
> >> the record is when it's *received* by the operator, not when it's
> processed
> >> by the operator.
> >> It seemed like there was the potential for confusion that
> >> "end-to-end-latency" might
> >> represent the latency from the event creation to the time the processor
> is
> >> done
> >> processing it.
> >>
> >> @John
> >> I'd rather err on the side of "not-enough" metrics as we can always add
> >> this to the
> >> stateless metrics later on. If we decide to measure the time at every
> node
> >> and don't
> >> find any evidence of a serious performance impact, and users indicate
> they
> >> would
> >> like to see this metric at all nodes, then we can easily start reporting
> >> them as well.
> >> WDYT?
> >>
> >> That said, sink nodes seem like a reasonable exception to the rule.
> >> Obviously users
> >> should be able to detect the time when the record reaches the output
> topic
> >> but that
> >> still leaves a gap in understanding how long the production latency was.
> >> This mirrors
> >> the consumption latency that is exposed by the task-level metrics, which
> >> are measured
> >> at the source node. For good symmetry what if we actually expose both
> the
> >> source
> >> and sink latency at the task-level? ie report both sets of statistical
> >> measurements with
> >> the additional tag -source/-sink
> >>
> >> @Bill
> >> Thanks for the comment about regarding the min! I hadn't considered that
> >> and it's
> >> quite useful to think about how and what is useful from a users point of
> >> view.
> >>
> >> Regarding your second. point, I'm inclined to leave that as an
> >> implementation detail
> >> but my take would be that the user should be allowed to control the
> record
> >> timestamp
> >> used for this with the timestamp extractor. My impression is that users
> may
> >> often embed
> >> the actual event time in the payload for whatever reason, and this
> >> represents the "true"
> >> timestamp as far as the Streams topology is concerned.
> >>
> >>
> >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com> wrote:
> >>
> >>> Thanks for the KIP, Sophie, this will be a useful metric to add.
> >>>
> >>> Regarding tracking min, I  think it could be valuable for users to
> >> discern
> >>> which part of their topologies are more efficient since this is a
> >>> task-level metric.  I realize everyone seems to be on board with
> >> including
> >>> min anyway, but I wanted to add my 2 cents on this topic should we
> decide
> >>> to revisit adding min or not.
> >>>
> >>> I do have a question regarding the calculation of staleness.
> >>> Is there going to be a consideration for timestamp extractors? Users
> >> could
> >>> prefer to use a timestamp embedded in the payload, and it could skew
> the
> >>> measurements.
> >>> I was wondering if we should specify in the KIP if setting the arrival
> >> time
> >>> is always going to come from the record timestamp, or is this an
> >>> implementation detail we can cover in the PR?
> >>>
> >>> Thanks!
> >>> Bill
> >>>
> >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
> >> wrote:
> >>>
> >>>> Thanks for the KIP Sophie.
> >>>>
> >>>> I think it's not useful to record the avg/mean; it sensitive to
> >>>> outliers. We should rather track the median (50th percentile).
> >>>>
> >>>> Not sure if tracking min is useful, but I am also ok to track it.
> >>>>
> >>>> However, I find it odd to track 75th percentile. Standard measures
> >> would
> >>>> the 90th or 95th -- I guess we don't need both, so maybe picking 90th
> >>>> might be more useful?
> >>>>
> >>>> About the name: "staleness" wound really odd, and if fact the metric
> >>>> does capture "latency" so we should call it "latency". I understand
> the
> >>>> issue that we already have a latency metric. So maybe we could call it
> >>>> `record-e2e-latency-*` ?
> >>>>
> >>>> While I agree that we should include out-or-order data (the KIP should
> >>>> talk about `out-of-order` data, not `late` data; data is only `late`
> if
> >>>> it's out-of-order and if it's dropped), I don't really understand why
> >>>> the new metric would help to configure grace period or retention time?
> >>>> As you mention in the KIP, both are define as max difference of
> >>>> `event-time - stream-time` and thus the new metric that takes
> >>>> system-/wallclock-time into account does not seem to help at all.
> >>>>
> >>>>
> >>>> Btw: there is a great talk about "How NOT to Measure Latency" by Gil
> >>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 5/14/20 7:17 PM, John Roesler wrote:
> >>>>> Hi Sophie,
> >>>>>
> >>>>> It seems like there would still be plenty of use cases for recording
> >>>>> this metric at all processors and not just stateful ones, but I'm
> >> happy
> >>>>> to suspend my arguments for now. Since you're proposing to keep
> >>>>> them at the processor-node level, it will be seamless later to add
> >>>>> in the stateless processors if we want. As a wise man once said,
> >>>>> "Adding is always easier than removing."
> >>>>>
> >>>>> Regarding the time measurement, it's an implementation detail
> >>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
> >>>>> prefer to measure the system time again when recording the
> >>>>> metric. I don't think we've seen any evidence that proves this
> >>>>> would harm performance, and the amount of inaccuracy using
> >>>>> the cached system time could incur is actually substantial. But,
> >>>>> if you want to just "not mention this" in the KIP, we can defer to
> >>>>> the actual PR discussion, at which time we're in a better position
> >>>>> to use benchmarks, etc., to make the call.
> >>>>>
> >>>>> Along the lines of the measurement accuracy discussion, one
> >>>>> minor thought I had is that maybe we should consider measuring
> >>>>> the task staleness metric at the sink, rather than the source, so
> >> that
> >>>>> it includes the processing latency of the task itself, not just the
> >>>> latency
> >>>>> of everything up to, but not including, the task (which seems
> >> confusing
> >>>>> for users). I guess this could also be an implementation detail,
> >>> though.
> >>>>>
> >>>>> Thanks for the update,
> >>>>> -John
> >>>>>
> >>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> >>>>>> Hey all,
> >>>>>>
> >>>>>> After discussing with Bruno I'd like to propose a small amendment,
> >>>>>> which is to record the processor-node-level metrics only for
> >>> *stateful*
> >>>>>> *operators*. They would still be considered a "processor-node-level"
> >>>>>> metric and not a "state-store-level" metric as the staleness is
> >> still
> >>>>>> a property of the node rather than of the state itself. However, it
> >>>> seems
> >>>>>> that this information is primarily useful for stateful operators
> >> that
> >>>> might
> >>>>>> be exposing state via IQ or otherwise dependent on the record time
> >>>>>> unlike a stateless operator.
> >>>>>>
> >>>>>> It's worth calling out that recent performance improvements to the
> >>>> metrics
> >>>>>> framework mean that we no longer fetch the system time at the
> >> operator
> >>>>>> level, but only once per task. In other words the system time is not
> >>>> updated
> >>>>>> between each process as a record flows through the subtopology, so
> >>>>>> debugging the processor-level latency via the stateleness will not
> >> be
> >>>>>> possible.Note that this doesn't mean the operator-level metrics are
> >>> not
> >>>>>> *useful* relative to the task-level metric. Upstream caching and/or
> >>>>>> suppression
> >>>>>> can still cause a record's staleness at some downstream stateful
> >>>> operator
> >>>>>> to deviate from the task-level staleness (recorded at the source
> >>> node).
> >>>>>>
> >>>>>> Please let me know if you have any concerns about this change. The
> >>>>>> KIP has been updated with the new proposal
> >>>>>>
> >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
> >>>> wrote:
> >>>>>>
> >>>>>>> Hi Sophie,
> >>>>>>>
> >>>>>>> Thank you for the KIP.
> >>>>>>>
> >>>>>>> The KIP looks good to me.
> >>>>>>>
> >>>>>>> 50th percentile:
> >>>>>>> I think we do not need it now. If we need it, we can add it. Here
> >> the
> >>>>>>> old truism applies: Adding is always easier than removing.
> >>>>>>>
> >>>>>>> processor-node-level metrics:
> >>>>>>> I think it is good to have the staleness metrics also on
> >>>>>>> processor-node-level. If we do not want to record them on all
> >>>>>>> processor nodes, you could restrict the recording to stateful
> >>>>>>> processor-nodes, since those are the ones that would benefit most
> >>> from
> >>>>>>> the staleness metrics.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Bruno
> >>>>>>>
> >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> >>>> sophie@confluent.io>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Yeah, the specific reason was just to align with the current
> >>> metrics.
> >>>>>>>>
> >>>>>>>> Is it better to conform than to be right? History has a lot to say
> >>> on
> >>>>>>> that
> >>>>>>>> matter
> >>>>>>>> but I'm not sure how much of it applies to the fine details of
> >>> metrics
> >>>>>>>> naming :P
> >>>>>>>>
> >>>>>>>> More seriously, I figured if people are looking at this metric
> >>> they're
> >>>>>>>> likely to
> >>>>>>>> be looking at all the others. Then naming this one "-mean" would
> >>>> probably
> >>>>>>>> lead some to conclude that the "-avg" suffix in the other metrics
> >>> has
> >>>> a
> >>>>>>>> different meaning.
> >>>>>>>>
> >>>>>>>> As for the percentiles, I actually like p99 (and p75) better. I'll
> >>>> swap
> >>>>>>>> that out
> >>>>>>>>
> >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vvcephei@apache.org
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks Sophie,
> >>>>>>>>>
> >>>>>>>>> I hope this isn't too nit-picky, but is there a reason to choose
> >>>> "avg"
> >>>>>>>>> instead
> >>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
> >> oversensitive
> >>>>>>> because
> >>>>>>>>> of the mistake I just made earlier, but it strikes me that "avg"
> >> is
> >>>>>>>>> actually
> >>>>>>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
> >>> is
> >>>>>>>>> specific.
> >>>>>>>>> I see other Kafka metrics with "avg", but none with "mean"; was
> >>> that
> >>>>>>> the
> >>>>>>>>> reason? If so, I'm +1.
> >>>>>>>>>
> >>>>>>>>> Regarding the names of the percentile, I actually couldn't find
> >>> _any_
> >>>>>>> other
> >>>>>>>>> metrics that use percentile. Was there a reason to choose "99th"
> >> as
> >>>>>>> opposed
> >>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm just
> >>>>>>> primarily
> >>>>>>>>> asking
> >>>>>>>>> for consistency's sake.
> >>>>>>>>>
> >>>>>>>>> Thanks again,
> >>>>>>>>> -John
> >>>>>>>>>
> >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> >>>>>>>>>> Alright, I can get behind adding the min metric for the sake of
> >>>>>>> pretty
> >>>>>>>>>> graphs
> >>>>>>>>>> (and trivial computation).
> >>>>>>>>>>
> >>>>>>>>>> I'm still on the fence regarding the mean (or 50th percentile)
> >>> but I
> >>>>>>> can
> >>>>>>>>> see
> >>>>>>>>>> how users might expect it and find it a bit disorienting not to
> >>>>>>> have. So
> >>>>>>>>> the
> >>>>>>>>>> updated proposed metrics are
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    - record-staleness-max [ms]
> >>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> >>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> >>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
> >>>>>>>>>>    - record-staleness-min [ms]
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> >> vvcephei@apache.org
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
> >> guess
> >>>>>>> the
> >>>>>>>>> mean
> >>>>>>>>>>> seems more interesting to me than the median, but neither are
> >> as
> >>>>>>>>>>> interesting as the higher percentiles (99th and max).
> >>>>>>>>>>>
> >>>>>>>>>>> Min isn’t really important for any SLAs, but it does round out
> >>> the
> >>>>>>>>> mental
> >>>>>>>>>>> picture of the distribution. I’ve always graphed min along with
> >>> the
> >>>>>>>>> other
> >>>>>>>>>>> metrics to help me understand how fast the system can be, which
> >>>>>>> helps
> >>>>>>>>> in
> >>>>>>>>>>> optimization decisions. It’s also a relatively inexpensive
> >> metric
> >>>>>>> to
> >>>>>>>>>>> compute, so it might be nice to just throw it in.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> >>>>>>>>>>>> G1:
> >>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to the
> >>>>>>> specific
> >>>>>>>>>>> task"
> >>>>>>>>>>>> but
> >>>>>>>>>>>> I'm happy with "record-staleness" if that drives the point
> >> home
> >>>>>>>>> better.
> >>>>>>>>>>> So
> >>>>>>>>>>>> it's the
> >>>>>>>>>>>> "staleness of the record when it is received by that task" --
> >>>>>>> will
> >>>>>>>>> update
> >>>>>>>>>>>> the KIP
> >>>>>>>>>>>>
> >>>>>>>>>>>> B1/J:
> >>>>>>>>>>>> I'm struggling to imagine a case where the min would actually
> >> be
> >>>>>>>>> useful,
> >>>>>>>>>>>> rather than
> >>>>>>>>>>>> just intellectually interesting. I don't feel strongly that we
> >>>>>>>>> shouldn't
> >>>>>>>>>>>> add it, but that's
> >>>>>>>>>>>> why I didn't include it from the start. Can you enlighten me
> >>>>>>> with an
> >>>>>>>>>>>> example?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I was also vaguely concerned about the overhead of adding
> >>>>>>> multiple
> >>>>>>>>>>>> percentile
> >>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
> >>> performance
> >>>>>>>>> hit we
> >>>>>>>>>>>> take on
> >>>>>>>>>>>> metrics computation?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile would be
> >>> the
> >>>>>>>>> median
> >>>>>>>>>>>> not the
> >>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
> >>>>>>> percentile, or
> >>>>>>>>>>> just
> >>>>>>>>>>>> one
> >>>>>>>>>>>> of the two?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks all!
> >>>>>>>>>>>> Sophie
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> >>>>>>> vvcephei@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Just some comments on the discussion so far:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> B2/G1:
> >>>>>>>>>>>>> In principle, it shouldn't matter whether we report "spans"
> >> or
> >>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
> >>>>>>> pretty
> >>>>>>>>>>>>> difficult to really measure (like time spent waiting in the
> >>>>>>>>> topics, or
> >>>>>>>>>>>>> time from the event happening to the ETL producer choosing to
> >>>>>>> send
> >>>>>>>>> it,
> >>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In other words, it's practically easier to compute spans by
> >>>>>>>>> subtracting
> >>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
> >>>>>>> spans.
> >>>>>>>>> You
> >>>>>>>>>>>>> can even just consider that the span computation from e2e
> >>>>>>> always
> >>>>>>>>> just
> >>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
> >> latency
> >>>>>>>>> from
> >>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the end
> >> you
> >>>>>>>>> care
> >>>>>>>>>>> about.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It seems like people really prefer to have spans when they
> >> are
> >>>>>>>>>>> debugging
> >>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
> >>>>>>> measurement
> >>>>>>>>>>>>> that basically every person/application cares about and
> >> should
> >>>>>>> be
> >>>>>>>>>>>>> monitoring.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Altogether, it really seem to provide more value to more
> >>>>>>> people if
> >>>>>>>>> we
> >>>>>>>>>>>>> report
> >>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
> >> think
> >>>>>>> I
> >>>>>>>>> have
> >>>>>>>>>>> no
> >>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> G2:
> >>>>>>>>>>>>> I think the processor-node metric is nice, since the inside
> >> of
> >>>>>>> a
> >>>>>>>>> task
> >>>>>>>>>>> can
> >>>>>>>>>>>>> introduce a significant amount of latency in some cases.
> >> Plus,
> >>>>>>>>> it's a
> >>>>>>>>>>> more
> >>>>>>>>>>>>> direct measurement, if you really wanted to know (for the
> >>>>>>> purposes
> >>>>>>>>> of
> >>>>>>>>>>> IQ
> >>>>>>>>>>>>> or something) how long it takes source events to "show up" at
> >>>>>>> the
> >>>>>>>>>>> store.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think actually recording it at every processor could be
> >>>>>>>>> expensive,
> >>>>>>>>>>> but we
> >>>>>>>>>>>>> already record a bunch of metrics at the node level.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> B1:
> >>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if it's
> >> a
> >>>>>>> poor
> >>>>>>>>>>> metric
> >>>>>>>>>>>>> for operational purposes, a lot of people might expect to see
> >>>>>>>>> "mean".
> >>>>>>>>>>>>> Actually,
> >>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to leave
> >> it
> >>>>>>>>> off?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I might suggest:
> >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> B3:
> >>>>>>>>>>>>> I agree we should include late records (though not the ones
> >> we
> >>>>>>>>> drop).
> >>>>>>>>>>>>> It may be spiky, but only when there are legitimately some
> >>>>>>> records
> >>>>>>>>>>> with a
> >>>>>>>>>>>>> high end-to-end latency, which is the whole point of these
> >>>>>>> metrics.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That's it! I don't think I have any other feedback, other
> >> than
> >>>>>>> a
> >>>>>>>>>>> request to
> >>>>>>>>>>>>> also report "min".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> -John
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
> >>>>>>> latency
> >>>>>>>>> of the
> >>>>>>>>>>>>> task
> >>>>>>>>>>>>>> and the latency spent sitting in intermediate topics. I have
> >>>>>>> a
> >>>>>>>>>>> similar
> >>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
> >>>>>>> a
> >>>>>>>>> task A
> >>>>>>>>>>>>>> actually measures the latency of the sub-topology up-to but
> >>>>>>> not
> >>>>>>>>>>> including
> >>>>>>>>>>>>>> the processing of A, which is a bit weird.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
> >>>>>>> since
> >>>>>>>>>>> today we
> >>>>>>>>>>>>>> already have several "latency" metrics already which are
> >>>>>>>>> measuring
> >>>>>>>>>>>>> elapsed
> >>>>>>>>>>>>>> system-time for processing a record / etc, while here we are
> >>>>>>>>>>> comparing
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> system wallclock time with the record timestamp.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> >>>>>>> (note we
> >>>>>>>>>>> already
> >>>>>>>>>>>>>> have a "record-lateness" metric), in which case recording at
> >>>>>>> the
> >>>>>>>>>>>>>> system-time before we start processing the record sounds
> >> more
> >>>>>>>>>>> natural.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
> >>>>>>> processor-node-level
> >>>>>>>>> DEBUG
> >>>>>>>>>>>>>> metric is worth to add, given that we already have a
> >>>>>>> task-level
> >>>>>>>>>>>>> processing
> >>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
> >>>>>>>>> similar
> >>>>>>>>>>> to
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> task-level e2e latency + task-level processing latency.
> >>>>>>>>> Personally I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>>> having a task-level record-staleness metric is sufficient.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> >>>>>>>>>>>>> sophie@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
> >>>>>>> this
> >>>>>>>>>>>>> specific
> >>>>>>>>>>>>>>> metric, as
> >>>>>>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
> >>>>>>> the
> >>>>>>>>>>> latency
> >>>>>>>>>>>>> you
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that would
> >>>>>>>>> hopefully
> >>>>>>>>>>>>> give a
> >>>>>>>>>>>>>>> good
> >>>>>>>>>>>>>>> sense of what *most* records will experience, and what
> >>>>>>> *close
> >>>>>>>>> to
> >>>>>>>>>>> all*
> >>>>>>>>>>>>>>> records
> >>>>>>>>>>>>>>> will.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> However I'm not married to these specific numbers and
> >>>>>>> could be
> >>>>>>>>>>>>> convinced.
> >>>>>>>>>>>>>>> Would be especially interested in hearing from users on
> >>>>>>> this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
> >>>>>>>>> this KIP
> >>>>>>>>>>>>> since
> >>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>> can always compute it themselves by subtracting the
> >>>>>>> previous
> >>>>>>>>> node's
> >>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way since
> >>>>>>>>> you can
> >>>>>>>>>>>>> always
> >>>>>>>>>>>>>>> compute one from the other, but I think the end-to-end
> >>>>>>> latency
> >>>>>>>>>>> feels
> >>>>>>>>>>>>> more
> >>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
> >>>>>>> bottlenecks
> >>>>>>>>> in the
> >>>>>>>>>>>>>>> topology but
> >>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to be
> >>>>>>>>> reflected
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> certain parts
> >>>>>>>>>>>>>>> of the topology. For example this might be useful for users
> >>>>>>>>> who are
> >>>>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>> roughly when a record that was just produced will be
> >>>>>>> included
> >>>>>>>>> in
> >>>>>>>>>>> their
> >>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>> results.
> >>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
> >>>>>>> make
> >>>>>>>>>>> that
> >>>>>>>>>>>>> clear
> >>>>>>>>>>>>>>> enough
> >>>>>>>>>>>>>>> in the KIP's motivation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> >>>>>>> short
> >>>>>>>>>>> answer is
> >>>>>>>>>>>>>>> "yes",
> >>>>>>>>>>>>>>> we will include late records. I added a paragraph to the
> >>>>>>> end
> >>>>>>>>> of the
> >>>>>>>>>>>>>>> Proposed
> >>>>>>>>>>>>>>> Changes section explaining the reasoning here, please let
> >>>>>>> me
> >>>>>>>>> know
> >>>>>>>>>>> if
> >>>>>>>>>>>>> you
> >>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>> any concerns.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
> >>>>>>>>>>> "process-latency",
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> metric
> >>>>>>>>>>>>>>> reflects the time for the literal Node#process method to
> >>>>>>> run
> >>>>>>>>>>> whereas
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> metric
> >>>>>>>>>>>>>>> would always be measured relative to the event timestamp.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> That said, the naming collision there is pretty confusing
> >>>>>>> so
> >>>>>>>>> I've
> >>>>>>>>>>>>> renamed
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
> >>>>>>> better
> >>>>>>>>>>>>> reflects
> >>>>>>>>>>>>>>> the nature
> >>>>>>>>>>>>>>> of the metric anyway.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> >>>>>>>>>>>>> reluctanthero104@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> >>>>>>>>> important
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>> understanding the bottleneck of the application.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> A couple of questions and ideas:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
> >>>>>>> max
> >>>>>>>>>>>>> percentiles?
> >>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> >>>>>>> well
> >>>>>>>>> in
> >>>>>>>>>>>>> production
> >>>>>>>>>>>>>>>> systems.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
> >>>>>>> if a
> >>>>>>>>>>> record
> >>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> >>>>>>> T(B->C) +
> >>>>>>>>>>> T(A->B) +
> >>>>>>>>>>>>>>> T(A)
> >>>>>>>>>>>>>>>> and so on, where P() represents the captured latency,
> >>>>>>> and T()
> >>>>>>>>>>>>> represents
> >>>>>>>>>>>>>>>> the time for transiting the records between two nodes,
> >>>>>>>>> including
> >>>>>>>>>>>>>>> processing
> >>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> >>>>>>>>> T(A->B)
> >>>>>>>>>>> are
> >>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> >>>>>>> there
> >>>>>>>>> is a
> >>>>>>>>>>>>> spike in
> >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> >>>>>>> time.
> >>>>>>>>> In the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> >>>>>>> record
> >>>>>>>>> exits
> >>>>>>>>>>>>> from the
> >>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
> >>>>>>> inside
> >>>>>>>>> the
> >>>>>>>>>>>>> funnel. Do
> >>>>>>>>>>>>>>>> you think we could have separate treatment for sink
> >>>>>>> nodes and
> >>>>>>>>>>> other
> >>>>>>>>>>>>>>>> nodes, so that other nodes only count the time receiving
> >>>>>>> the
> >>>>>>>>>>> record
> >>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
> >>>>>>>>> discuss
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> alternative to see if it is reasonable.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> >>>>>>> well,
> >>>>>>>>> they
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>> create some really spiky graphs when the out-of-order
> >>>>>>>>> records are
> >>>>>>>>>>>>>>>> interleaving with on time records. Should we also supply
> >>>>>>> a
> >>>>>>>>> smooth
> >>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>> of the latency metrics, or user should just take care of
> >>>>>>> it
> >>>>>>>>> by
> >>>>>>>>>>>>> themself?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
> >>>>>>>>> relation
> >>>>>>>>>>>>> with our
> >>>>>>>>>>>>>>>> existing processing latency metrics, could you add some
> >>>>>>>>> context
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
> >>>>>>> the
> >>>>>>>>>>> best?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Boyang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> >>>>>>>>>>>>> sophie@confluent.io
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> >>>>>>> to
> >>>>>>>>> add
> >>>>>>>>>>>>> end-to-end
> >>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Sophie
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by "Matthias J. Sax" <mj...@apache.org>.
I am also happy with max/min/99/90. And I buy your naming argument about
staleness vs latency.


-Matthias

On 5/15/20 12:24 PM, Boyang Chen wrote:
> Hey Sophie,
> 
> 90/99/min/max make sense to me.
> 
> On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
> 
>> @Matthias
>> Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark of
>> the video
>> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
>>
>> And that's what he says about the 95th percentile! Imagine what he would
>> say about
>> the 50th :P
>>
>> But seriously, since we can't seem to agree that the mean or 50th
>> percentile is actually
>> useful I'm inclined to resurrect my original proposal, neither. But I think
>> that's a good
>> argument against the 75th, which I admittedly chose somewhat arbitrarily as
>> an
>> intermediate between the 50th and the higher percentiles. How about:
>>
>> -max
>> -p99
>> -p90
>> -min
>>
>> with p50/mean still up for debate if anyone feels strongly for either of
>> them.
>>
>> Regarding the name, I've already flip-flopped on this so I'm definitely
>> still open to
>> further arguments. But the reason for changing it from end-to-end-latency
>> (which
>> is similar to what you propose) is that this metric technically reflects
>> how old (ie how "stale")
>> the record is when it's *received* by the operator, not when it's processed
>> by the operator.
>> It seemed like there was the potential for confusion that
>> "end-to-end-latency" might
>> represent the latency from the event creation to the time the processor is
>> done
>> processing it.
>>
>> @John
>> I'd rather err on the side of "not-enough" metrics as we can always add
>> this to the
>> stateless metrics later on. If we decide to measure the time at every node
>> and don't
>> find any evidence of a serious performance impact, and users indicate they
>> would
>> like to see this metric at all nodes, then we can easily start reporting
>> them as well.
>> WDYT?
>>
>> That said, sink nodes seem like a reasonable exception to the rule.
>> Obviously users
>> should be able to detect the time when the record reaches the output topic
>> but that
>> still leaves a gap in understanding how long the production latency was.
>> This mirrors
>> the consumption latency that is exposed by the task-level metrics, which
>> are measured
>> at the source node. For good symmetry what if we actually expose both the
>> source
>> and sink latency at the task-level? ie report both sets of statistical
>> measurements with
>> the additional tag -source/-sink
>>
>> @Bill
>> Thanks for the comment about regarding the min! I hadn't considered that
>> and it's
>> quite useful to think about how and what is useful from a users point of
>> view.
>>
>> Regarding your second. point, I'm inclined to leave that as an
>> implementation detail
>> but my take would be that the user should be allowed to control the record
>> timestamp
>> used for this with the timestamp extractor. My impression is that users may
>> often embed
>> the actual event time in the payload for whatever reason, and this
>> represents the "true"
>> timestamp as far as the Streams topology is concerned.
>>
>>
>> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com> wrote:
>>
>>> Thanks for the KIP, Sophie, this will be a useful metric to add.
>>>
>>> Regarding tracking min, I  think it could be valuable for users to
>> discern
>>> which part of their topologies are more efficient since this is a
>>> task-level metric.  I realize everyone seems to be on board with
>> including
>>> min anyway, but I wanted to add my 2 cents on this topic should we decide
>>> to revisit adding min or not.
>>>
>>> I do have a question regarding the calculation of staleness.
>>> Is there going to be a consideration for timestamp extractors? Users
>> could
>>> prefer to use a timestamp embedded in the payload, and it could skew the
>>> measurements.
>>> I was wondering if we should specify in the KIP if setting the arrival
>> time
>>> is always going to come from the record timestamp, or is this an
>>> implementation detail we can cover in the PR?
>>>
>>> Thanks!
>>> Bill
>>>
>>> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
>> wrote:
>>>
>>>> Thanks for the KIP Sophie.
>>>>
>>>> I think it's not useful to record the avg/mean; it sensitive to
>>>> outliers. We should rather track the median (50th percentile).
>>>>
>>>> Not sure if tracking min is useful, but I am also ok to track it.
>>>>
>>>> However, I find it odd to track 75th percentile. Standard measures
>> would
>>>> the 90th or 95th -- I guess we don't need both, so maybe picking 90th
>>>> might be more useful?
>>>>
>>>> About the name: "staleness" wound really odd, and if fact the metric
>>>> does capture "latency" so we should call it "latency". I understand the
>>>> issue that we already have a latency metric. So maybe we could call it
>>>> `record-e2e-latency-*` ?
>>>>
>>>> While I agree that we should include out-or-order data (the KIP should
>>>> talk about `out-of-order` data, not `late` data; data is only `late` if
>>>> it's out-of-order and if it's dropped), I don't really understand why
>>>> the new metric would help to configure grace period or retention time?
>>>> As you mention in the KIP, both are define as max difference of
>>>> `event-time - stream-time` and thus the new metric that takes
>>>> system-/wallclock-time into account does not seem to help at all.
>>>>
>>>>
>>>> Btw: there is a great talk about "How NOT to Measure Latency" by Gil
>>>> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 5/14/20 7:17 PM, John Roesler wrote:
>>>>> Hi Sophie,
>>>>>
>>>>> It seems like there would still be plenty of use cases for recording
>>>>> this metric at all processors and not just stateful ones, but I'm
>> happy
>>>>> to suspend my arguments for now. Since you're proposing to keep
>>>>> them at the processor-node level, it will be seamless later to add
>>>>> in the stateless processors if we want. As a wise man once said,
>>>>> "Adding is always easier than removing."
>>>>>
>>>>> Regarding the time measurement, it's an implementation detail
>>>>> we don't need to consider in the KIP. Nevertheless, I'd greatly
>>>>> prefer to measure the system time again when recording the
>>>>> metric. I don't think we've seen any evidence that proves this
>>>>> would harm performance, and the amount of inaccuracy using
>>>>> the cached system time could incur is actually substantial. But,
>>>>> if you want to just "not mention this" in the KIP, we can defer to
>>>>> the actual PR discussion, at which time we're in a better position
>>>>> to use benchmarks, etc., to make the call.
>>>>>
>>>>> Along the lines of the measurement accuracy discussion, one
>>>>> minor thought I had is that maybe we should consider measuring
>>>>> the task staleness metric at the sink, rather than the source, so
>> that
>>>>> it includes the processing latency of the task itself, not just the
>>>> latency
>>>>> of everything up to, but not including, the task (which seems
>> confusing
>>>>> for users). I guess this could also be an implementation detail,
>>> though.
>>>>>
>>>>> Thanks for the update,
>>>>> -John
>>>>>
>>>>> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
>>>>>> Hey all,
>>>>>>
>>>>>> After discussing with Bruno I'd like to propose a small amendment,
>>>>>> which is to record the processor-node-level metrics only for
>>> *stateful*
>>>>>> *operators*. They would still be considered a "processor-node-level"
>>>>>> metric and not a "state-store-level" metric as the staleness is
>> still
>>>>>> a property of the node rather than of the state itself. However, it
>>>> seems
>>>>>> that this information is primarily useful for stateful operators
>> that
>>>> might
>>>>>> be exposing state via IQ or otherwise dependent on the record time
>>>>>> unlike a stateless operator.
>>>>>>
>>>>>> It's worth calling out that recent performance improvements to the
>>>> metrics
>>>>>> framework mean that we no longer fetch the system time at the
>> operator
>>>>>> level, but only once per task. In other words the system time is not
>>>> updated
>>>>>> between each process as a record flows through the subtopology, so
>>>>>> debugging the processor-level latency via the stateleness will not
>> be
>>>>>> possible.Note that this doesn't mean the operator-level metrics are
>>> not
>>>>>> *useful* relative to the task-level metric. Upstream caching and/or
>>>>>> suppression
>>>>>> can still cause a record's staleness at some downstream stateful
>>>> operator
>>>>>> to deviate from the task-level staleness (recorded at the source
>>> node).
>>>>>>
>>>>>> Please let me know if you have any concerns about this change. The
>>>>>> KIP has been updated with the new proposal
>>>>>>
>>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
>>>> wrote:
>>>>>>
>>>>>>> Hi Sophie,
>>>>>>>
>>>>>>> Thank you for the KIP.
>>>>>>>
>>>>>>> The KIP looks good to me.
>>>>>>>
>>>>>>> 50th percentile:
>>>>>>> I think we do not need it now. If we need it, we can add it. Here
>> the
>>>>>>> old truism applies: Adding is always easier than removing.
>>>>>>>
>>>>>>> processor-node-level metrics:
>>>>>>> I think it is good to have the staleness metrics also on
>>>>>>> processor-node-level. If we do not want to record them on all
>>>>>>> processor nodes, you could restrict the recording to stateful
>>>>>>> processor-nodes, since those are the ones that would benefit most
>>> from
>>>>>>> the staleness metrics.
>>>>>>>
>>>>>>> Best,
>>>>>>> Bruno
>>>>>>>
>>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
>>>> sophie@confluent.io>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Yeah, the specific reason was just to align with the current
>>> metrics.
>>>>>>>>
>>>>>>>> Is it better to conform than to be right? History has a lot to say
>>> on
>>>>>>> that
>>>>>>>> matter
>>>>>>>> but I'm not sure how much of it applies to the fine details of
>>> metrics
>>>>>>>> naming :P
>>>>>>>>
>>>>>>>> More seriously, I figured if people are looking at this metric
>>> they're
>>>>>>>> likely to
>>>>>>>> be looking at all the others. Then naming this one "-mean" would
>>>> probably
>>>>>>>> lead some to conclude that the "-avg" suffix in the other metrics
>>> has
>>>> a
>>>>>>>> different meaning.
>>>>>>>>
>>>>>>>> As for the percentiles, I actually like p99 (and p75) better. I'll
>>>> swap
>>>>>>>> that out
>>>>>>>>
>>>>>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vvcephei@apache.org
>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Sophie,
>>>>>>>>>
>>>>>>>>> I hope this isn't too nit-picky, but is there a reason to choose
>>>> "avg"
>>>>>>>>> instead
>>>>>>>>> of "mean"? Maybe this is too paranoid, and I might be
>> oversensitive
>>>>>>> because
>>>>>>>>> of the mistake I just made earlier, but it strikes me that "avg"
>> is
>>>>>>>>> actually
>>>>>>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
>>> is
>>>>>>>>> specific.
>>>>>>>>> I see other Kafka metrics with "avg", but none with "mean"; was
>>> that
>>>>>>> the
>>>>>>>>> reason? If so, I'm +1.
>>>>>>>>>
>>>>>>>>> Regarding the names of the percentile, I actually couldn't find
>>> _any_
>>>>>>> other
>>>>>>>>> metrics that use percentile. Was there a reason to choose "99th"
>> as
>>>>>>> opposed
>>>>>>>>> to "p99" or any other scheme? This is not a criticism, I'm just
>>>>>>> primarily
>>>>>>>>> asking
>>>>>>>>> for consistency's sake.
>>>>>>>>>
>>>>>>>>> Thanks again,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
>>>>>>>>>> Alright, I can get behind adding the min metric for the sake of
>>>>>>> pretty
>>>>>>>>>> graphs
>>>>>>>>>> (and trivial computation).
>>>>>>>>>>
>>>>>>>>>> I'm still on the fence regarding the mean (or 50th percentile)
>>> but I
>>>>>>> can
>>>>>>>>> see
>>>>>>>>>> how users might expect it and find it a bit disorienting not to
>>>>>>> have. So
>>>>>>>>> the
>>>>>>>>>> updated proposed metrics are
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    - record-staleness-max [ms]
>>>>>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
>>>>>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
>>>>>>>>>>    - record-staleness-avg [ms] *(mean)*
>>>>>>>>>>    - record-staleness-min [ms]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
>> vvcephei@apache.org
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
>> guess
>>>>>>> the
>>>>>>>>> mean
>>>>>>>>>>> seems more interesting to me than the median, but neither are
>> as
>>>>>>>>>>> interesting as the higher percentiles (99th and max).
>>>>>>>>>>>
>>>>>>>>>>> Min isn’t really important for any SLAs, but it does round out
>>> the
>>>>>>>>> mental
>>>>>>>>>>> picture of the distribution. I’ve always graphed min along with
>>> the
>>>>>>>>> other
>>>>>>>>>>> metrics to help me understand how fast the system can be, which
>>>>>>> helps
>>>>>>>>> in
>>>>>>>>>>> optimization decisions. It’s also a relatively inexpensive
>> metric
>>>>>>> to
>>>>>>>>>>> compute, so it might be nice to just throw it in.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
>>>>>>>>>>>> G1:
>>>>>>>>>>>> I was considering it as the "end-to-end latency *up* to the
>>>>>>> specific
>>>>>>>>>>> task"
>>>>>>>>>>>> but
>>>>>>>>>>>> I'm happy with "record-staleness" if that drives the point
>> home
>>>>>>>>> better.
>>>>>>>>>>> So
>>>>>>>>>>>> it's the
>>>>>>>>>>>> "staleness of the record when it is received by that task" --
>>>>>>> will
>>>>>>>>> update
>>>>>>>>>>>> the KIP
>>>>>>>>>>>>
>>>>>>>>>>>> B1/J:
>>>>>>>>>>>> I'm struggling to imagine a case where the min would actually
>> be
>>>>>>>>> useful,
>>>>>>>>>>>> rather than
>>>>>>>>>>>> just intellectually interesting. I don't feel strongly that we
>>>>>>>>> shouldn't
>>>>>>>>>>>> add it, but that's
>>>>>>>>>>>> why I didn't include it from the start. Can you enlighten me
>>>>>>> with an
>>>>>>>>>>>> example?
>>>>>>>>>>>>
>>>>>>>>>>>> I was also vaguely concerned about the overhead of adding
>>>>>>> multiple
>>>>>>>>>>>> percentile
>>>>>>>>>>>> metrics. Do we have any data to indicate what kind of
>>> performance
>>>>>>>>> hit we
>>>>>>>>>>>> take on
>>>>>>>>>>>> metrics computation?
>>>>>>>>>>>>
>>>>>>>>>>>> Also, not to be too pedantic but the 50th percentile would be
>>> the
>>>>>>>>> median
>>>>>>>>>>>> not the
>>>>>>>>>>>> mean. Would you propose to add the mean *and* the 50th
>>>>>>> percentile, or
>>>>>>>>>>> just
>>>>>>>>>>>> one
>>>>>>>>>>>> of the two?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks all!
>>>>>>>>>>>> Sophie
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
>>>>>>> vvcephei@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just some comments on the discussion so far:
>>>>>>>>>>>>>
>>>>>>>>>>>>> B2/G1:
>>>>>>>>>>>>> In principle, it shouldn't matter whether we report "spans"
>> or
>>>>>>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
>>>>>>> pretty
>>>>>>>>>>>>> difficult to really measure (like time spent waiting in the
>>>>>>>>> topics, or
>>>>>>>>>>>>> time from the event happening to the ETL producer choosing to
>>>>>>> send
>>>>>>>>> it,
>>>>>>>>>>>>> or time spent in send/receive buffers, etc., etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In other words, it's practically easier to compute spans by
>>>>>>>>> subtracting
>>>>>>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
>>>>>>> spans.
>>>>>>>>> You
>>>>>>>>>>>>> can even just consider that the span computation from e2e
>>>>>>> always
>>>>>>>>> just
>>>>>>>>>>>>> involves subtracting two numbers, whereas computing e2e
>> latency
>>>>>>>>> from
>>>>>>>>>>>>> spans involves adding _all_ the spans leading up to the end
>> you
>>>>>>>>> care
>>>>>>>>>>> about.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems like people really prefer to have spans when they
>> are
>>>>>>>>>>> debugging
>>>>>>>>>>>>> latency problems, whereas e2e latency is a more general
>>>>>>> measurement
>>>>>>>>>>>>> that basically every person/application cares about and
>> should
>>>>>>> be
>>>>>>>>>>>>> monitoring.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Altogether, it really seem to provide more value to more
>>>>>>> people if
>>>>>>>>> we
>>>>>>>>>>>>> report
>>>>>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
>> think
>>>>>>> I
>>>>>>>>> have
>>>>>>>>>>> no
>>>>>>>>>>>>> preference, I'd defer to other peoples' intuition.
>>>>>>>>>>>>>
>>>>>>>>>>>>> G2:
>>>>>>>>>>>>> I think the processor-node metric is nice, since the inside
>> of
>>>>>>> a
>>>>>>>>> task
>>>>>>>>>>> can
>>>>>>>>>>>>> introduce a significant amount of latency in some cases.
>> Plus,
>>>>>>>>> it's a
>>>>>>>>>>> more
>>>>>>>>>>>>> direct measurement, if you really wanted to know (for the
>>>>>>> purposes
>>>>>>>>> of
>>>>>>>>>>> IQ
>>>>>>>>>>>>> or something) how long it takes source events to "show up" at
>>>>>>> the
>>>>>>>>>>> store.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think actually recording it at every processor could be
>>>>>>>>> expensive,
>>>>>>>>>>> but we
>>>>>>>>>>>>> already record a bunch of metrics at the node level.
>>>>>>>>>>>>>
>>>>>>>>>>>>> B1:
>>>>>>>>>>>>> I think 50% could be reasonable to record also. Even if it's
>> a
>>>>>>> poor
>>>>>>>>>>> metric
>>>>>>>>>>>>> for operational purposes, a lot of people might expect to see
>>>>>>>>> "mean".
>>>>>>>>>>>>> Actually,
>>>>>>>>>>>>> I was surprised not to see "min". Is there a reason to leave
>> it
>>>>>>>>> off?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I might suggest:
>>>>>>>>>>>>> min, mean (50th), 75th, 99th, max
>>>>>>>>>>>>>
>>>>>>>>>>>>> B3:
>>>>>>>>>>>>> I agree we should include late records (though not the ones
>> we
>>>>>>>>> drop).
>>>>>>>>>>>>> It may be spiky, but only when there are legitimately some
>>>>>>> records
>>>>>>>>>>> with a
>>>>>>>>>>>>> high end-to-end latency, which is the whole point of these
>>>>>>> metrics.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's it! I don't think I have any other feedback, other
>> than
>>>>>>> a
>>>>>>>>>>> request to
>>>>>>>>>>>>> also report "min".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
>>>>>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) The end-to-end latency includes both the processing
>>>>>>> latency
>>>>>>>>> of the
>>>>>>>>>>>>> task
>>>>>>>>>>>>>> and the latency spent sitting in intermediate topics. I have
>>>>>>> a
>>>>>>>>>>> similar
>>>>>>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
>>>>>>> a
>>>>>>>>> task A
>>>>>>>>>>>>>> actually measures the latency of the sub-topology up-to but
>>>>>>> not
>>>>>>>>>>> including
>>>>>>>>>>>>>> the processing of A, which is a bit weird.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
>>>>>>> since
>>>>>>>>>>> today we
>>>>>>>>>>>>>> already have several "latency" metrics already which are
>>>>>>>>> measuring
>>>>>>>>>>>>> elapsed
>>>>>>>>>>>>>> system-time for processing a record / etc, while here we are
>>>>>>>>>>> comparing
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> system wallclock time with the record timestamp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
>>>>>>> (note we
>>>>>>>>>>> already
>>>>>>>>>>>>>> have a "record-lateness" metric), in which case recording at
>>>>>>> the
>>>>>>>>>>>>>> system-time before we start processing the record sounds
>> more
>>>>>>>>>>> natural.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) With that in mind, I'm wondering if the
>>>>>>> processor-node-level
>>>>>>>>> DEBUG
>>>>>>>>>>>>>> metric is worth to add, given that we already have a
>>>>>>> task-level
>>>>>>>>>>>>> processing
>>>>>>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
>>>>>>>>> similar
>>>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> task-level e2e latency + task-level processing latency.
>>>>>>>>> Personally I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>> having a task-level record-staleness metric is sufficient.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
>>>>>>>>>>>>> sophie@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
>>>>>>> this
>>>>>>>>>>>>> specific
>>>>>>>>>>>>>>> metric, as
>>>>>>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
>>>>>>> the
>>>>>>>>>>> latency
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> reasonably expect to see. I chose percentiles that would
>>>>>>>>> hopefully
>>>>>>>>>>>>> give a
>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>> sense of what *most* records will experience, and what
>>>>>>> *close
>>>>>>>>> to
>>>>>>>>>>> all*
>>>>>>>>>>>>>>> records
>>>>>>>>>>>>>>> will.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However I'm not married to these specific numbers and
>>>>>>> could be
>>>>>>>>>>>>> convinced.
>>>>>>>>>>>>>>> Would be especially interested in hearing from users on
>>>>>>> this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
>>>>>>>>> this KIP
>>>>>>>>>>>>> since
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>> can always compute it themselves by subtracting the
>>>>>>> previous
>>>>>>>>> node's
>>>>>>>>>>>>>>> end-to-end latency. I guess we could do it either way since
>>>>>>>>> you can
>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> compute one from the other, but I think the end-to-end
>>>>>>> latency
>>>>>>>>>>> feels
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> valuable as it's main motivation is not to debug
>>>>>>> bottlenecks
>>>>>>>>> in the
>>>>>>>>>>>>>>> topology but
>>>>>>>>>>>>>>> to give users a sense of how long it takes arecord to be
>>>>>>>>> reflected
>>>>>>>>>>> in
>>>>>>>>>>>>>>> certain parts
>>>>>>>>>>>>>>> of the topology. For example this might be useful for users
>>>>>>>>> who are
>>>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>> roughly when a record that was just produced will be
>>>>>>> included
>>>>>>>>> in
>>>>>>>>>>> their
>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
>>>>>>> make
>>>>>>>>>>> that
>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>> enough
>>>>>>>>>>>>>>> in the KIP's motivation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
>>>>>>> short
>>>>>>>>>>> answer is
>>>>>>>>>>>>>>> "yes",
>>>>>>>>>>>>>>> we will include late records. I added a paragraph to the
>>>>>>> end
>>>>>>>>> of the
>>>>>>>>>>>>>>> Proposed
>>>>>>>>>>>>>>> Changes section explaining the reasoning here, please let
>>>>>>> me
>>>>>>>>> know
>>>>>>>>>>> if
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> any concerns.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4. Assuming you're referring to the existing metric
>>>>>>>>>>> "process-latency",
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> metric
>>>>>>>>>>>>>>> reflects the time for the literal Node#process method to
>>>>>>> run
>>>>>>>>>>> whereas
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> metric
>>>>>>>>>>>>>>> would always be measured relative to the event timestamp.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, the naming collision there is pretty confusing
>>>>>>> so
>>>>>>>>> I've
>>>>>>>>>>>>> renamed
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
>>>>>>> better
>>>>>>>>>>>>> reflects
>>>>>>>>>>>>>>> the nature
>>>>>>>>>>>>>>> of the metric anyway.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
>>>>>>>>>>>>> reluctanthero104@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
>>>>>>>>> important
>>>>>>>>>>> for
>>>>>>>>>>>>>>>> understanding the bottleneck of the application.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A couple of questions and ideas:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
>>>>>>> max
>>>>>>>>>>>>> percentiles?
>>>>>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
>>>>>>> well
>>>>>>>>> in
>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>> systems.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
>>>>>>> if a
>>>>>>>>>>> record
>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
>>>>>>> T(B->C) +
>>>>>>>>>>> T(A->B) +
>>>>>>>>>>>>>>> T(A)
>>>>>>>>>>>>>>>> and so on, where P() represents the captured latency,
>>>>>>> and T()
>>>>>>>>>>>>> represents
>>>>>>>>>>>>>>>> the time for transiting the records between two nodes,
>>>>>>>>> including
>>>>>>>>>>>>>>> processing
>>>>>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
>>>>>>>>> T(A->B)
>>>>>>>>>>> are
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
>>>>>>> there
>>>>>>>>> is a
>>>>>>>>>>>>> spike in
>>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
>>>>>>> time.
>>>>>>>>> In the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
>>>>>>> record
>>>>>>>>> exits
>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>> sink as this marks the whole time this record spent
>>>>>>> inside
>>>>>>>>> the
>>>>>>>>>>>>> funnel. Do
>>>>>>>>>>>>>>>> you think we could have separate treatment for sink
>>>>>>> nodes and
>>>>>>>>>>> other
>>>>>>>>>>>>>>>> nodes, so that other nodes only count the time receiving
>>>>>>> the
>>>>>>>>>>> record
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
>>>>>>>>> discuss
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> alternative to see if it is reasonable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
>>>>>>> well,
>>>>>>>>> they
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> create some really spiky graphs when the out-of-order
>>>>>>>>> records are
>>>>>>>>>>>>>>>> interleaving with on time records. Should we also supply
>>>>>>> a
>>>>>>>>> smooth
>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>> of the latency metrics, or user should just take care of
>>>>>>> it
>>>>>>>>> by
>>>>>>>>>>>>> themself?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
>>>>>>>>> relation
>>>>>>>>>>>>> with our
>>>>>>>>>>>>>>>> existing processing latency metrics, could you add some
>>>>>>>>> context
>>>>>>>>>>> on
>>>>>>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
>>>>>>> the
>>>>>>>>>>> best?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Boyang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
>>>>>>>>>>>>> sophie@confluent.io
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
>>>>>>> to
>>>>>>>>> add
>>>>>>>>>>>>> end-to-end
>>>>>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Sophie
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
> 


Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Boyang Chen <re...@gmail.com>.
Hey Sophie,

90/99/min/max make sense to me.

On Fri, May 15, 2020 at 12:20 PM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> @Matthias
> Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark of
> the video
> *you* linked: https://youtu.be/lJ8ydIuPFeU?t=293
>
> And that's what he says about the 95th percentile! Imagine what he would
> say about
> the 50th :P
>
> But seriously, since we can't seem to agree that the mean or 50th
> percentile is actually
> useful I'm inclined to resurrect my original proposal, neither. But I think
> that's a good
> argument against the 75th, which I admittedly chose somewhat arbitrarily as
> an
> intermediate between the 50th and the higher percentiles. How about:
>
> -max
> -p99
> -p90
> -min
>
> with p50/mean still up for debate if anyone feels strongly for either of
> them.
>
> Regarding the name, I've already flip-flopped on this so I'm definitely
> still open to
> further arguments. But the reason for changing it from end-to-end-latency
> (which
> is similar to what you propose) is that this metric technically reflects
> how old (ie how "stale")
> the record is when it's *received* by the operator, not when it's processed
> by the operator.
> It seemed like there was the potential for confusion that
> "end-to-end-latency" might
> represent the latency from the event creation to the time the processor is
> done
> processing it.
>
> @John
> I'd rather err on the side of "not-enough" metrics as we can always add
> this to the
> stateless metrics later on. If we decide to measure the time at every node
> and don't
> find any evidence of a serious performance impact, and users indicate they
> would
> like to see this metric at all nodes, then we can easily start reporting
> them as well.
> WDYT?
>
> That said, sink nodes seem like a reasonable exception to the rule.
> Obviously users
> should be able to detect the time when the record reaches the output topic
> but that
> still leaves a gap in understanding how long the production latency was.
> This mirrors
> the consumption latency that is exposed by the task-level metrics, which
> are measured
> at the source node. For good symmetry what if we actually expose both the
> source
> and sink latency at the task-level? ie report both sets of statistical
> measurements with
> the additional tag -source/-sink
>
> @Bill
> Thanks for the comment about regarding the min! I hadn't considered that
> and it's
> quite useful to think about how and what is useful from a users point of
> view.
>
> Regarding your second. point, I'm inclined to leave that as an
> implementation detail
> but my take would be that the user should be allowed to control the record
> timestamp
> used for this with the timestamp extractor. My impression is that users may
> often embed
> the actual event time in the payload for whatever reason, and this
> represents the "true"
> timestamp as far as the Streams topology is concerned.
>
>
> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com> wrote:
>
> > Thanks for the KIP, Sophie, this will be a useful metric to add.
> >
> > Regarding tracking min, I  think it could be valuable for users to
> discern
> > which part of their topologies are more efficient since this is a
> > task-level metric.  I realize everyone seems to be on board with
> including
> > min anyway, but I wanted to add my 2 cents on this topic should we decide
> > to revisit adding min or not.
> >
> > I do have a question regarding the calculation of staleness.
> > Is there going to be a consideration for timestamp extractors? Users
> could
> > prefer to use a timestamp embedded in the payload, and it could skew the
> > measurements.
> > I was wondering if we should specify in the KIP if setting the arrival
> time
> > is always going to come from the record timestamp, or is this an
> > implementation detail we can cover in the PR?
> >
> > Thanks!
> > Bill
> >
> > On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > Thanks for the KIP Sophie.
> > >
> > > I think it's not useful to record the avg/mean; it sensitive to
> > > outliers. We should rather track the median (50th percentile).
> > >
> > > Not sure if tracking min is useful, but I am also ok to track it.
> > >
> > > However, I find it odd to track 75th percentile. Standard measures
> would
> > > the 90th or 95th -- I guess we don't need both, so maybe picking 90th
> > > might be more useful?
> > >
> > > About the name: "staleness" wound really odd, and if fact the metric
> > > does capture "latency" so we should call it "latency". I understand the
> > > issue that we already have a latency metric. So maybe we could call it
> > > `record-e2e-latency-*` ?
> > >
> > > While I agree that we should include out-or-order data (the KIP should
> > > talk about `out-of-order` data, not `late` data; data is only `late` if
> > > it's out-of-order and if it's dropped), I don't really understand why
> > > the new metric would help to configure grace period or retention time?
> > > As you mention in the KIP, both are define as max difference of
> > > `event-time - stream-time` and thus the new metric that takes
> > > system-/wallclock-time into account does not seem to help at all.
> > >
> > >
> > > Btw: there is a great talk about "How NOT to Measure Latency" by Gil
> > > Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 5/14/20 7:17 PM, John Roesler wrote:
> > > > Hi Sophie,
> > > >
> > > > It seems like there would still be plenty of use cases for recording
> > > > this metric at all processors and not just stateful ones, but I'm
> happy
> > > > to suspend my arguments for now. Since you're proposing to keep
> > > > them at the processor-node level, it will be seamless later to add
> > > > in the stateless processors if we want. As a wise man once said,
> > > > "Adding is always easier than removing."
> > > >
> > > > Regarding the time measurement, it's an implementation detail
> > > > we don't need to consider in the KIP. Nevertheless, I'd greatly
> > > > prefer to measure the system time again when recording the
> > > > metric. I don't think we've seen any evidence that proves this
> > > > would harm performance, and the amount of inaccuracy using
> > > > the cached system time could incur is actually substantial. But,
> > > > if you want to just "not mention this" in the KIP, we can defer to
> > > > the actual PR discussion, at which time we're in a better position
> > > > to use benchmarks, etc., to make the call.
> > > >
> > > > Along the lines of the measurement accuracy discussion, one
> > > > minor thought I had is that maybe we should consider measuring
> > > > the task staleness metric at the sink, rather than the source, so
> that
> > > > it includes the processing latency of the task itself, not just the
> > > latency
> > > > of everything up to, but not including, the task (which seems
> confusing
> > > > for users). I guess this could also be an implementation detail,
> > though.
> > > >
> > > > Thanks for the update,
> > > > -John
> > > >
> > > > On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > > >> Hey all,
> > > >>
> > > >> After discussing with Bruno I'd like to propose a small amendment,
> > > >> which is to record the processor-node-level metrics only for
> > *stateful*
> > > >> *operators*. They would still be considered a "processor-node-level"
> > > >> metric and not a "state-store-level" metric as the staleness is
> still
> > > >> a property of the node rather than of the state itself. However, it
> > > seems
> > > >> that this information is primarily useful for stateful operators
> that
> > > might
> > > >> be exposing state via IQ or otherwise dependent on the record time
> > > >> unlike a stateless operator.
> > > >>
> > > >> It's worth calling out that recent performance improvements to the
> > > metrics
> > > >> framework mean that we no longer fetch the system time at the
> operator
> > > >> level, but only once per task. In other words the system time is not
> > > updated
> > > >> between each process as a record flows through the subtopology, so
> > > >> debugging the processor-level latency via the stateleness will not
> be
> > > >> possible.Note that this doesn't mean the operator-level metrics are
> > not
> > > >> *useful* relative to the task-level metric. Upstream caching and/or
> > > >> suppression
> > > >> can still cause a record's staleness at some downstream stateful
> > > operator
> > > >> to deviate from the task-level staleness (recorded at the source
> > node).
> > > >>
> > > >> Please let me know if you have any concerns about this change. The
> > > >> KIP has been updated with the new proposal
> > > >>
> > > >> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
> > > wrote:
> > > >>
> > > >>> Hi Sophie,
> > > >>>
> > > >>> Thank you for the KIP.
> > > >>>
> > > >>> The KIP looks good to me.
> > > >>>
> > > >>> 50th percentile:
> > > >>> I think we do not need it now. If we need it, we can add it. Here
> the
> > > >>> old truism applies: Adding is always easier than removing.
> > > >>>
> > > >>> processor-node-level metrics:
> > > >>> I think it is good to have the staleness metrics also on
> > > >>> processor-node-level. If we do not want to record them on all
> > > >>> processor nodes, you could restrict the recording to stateful
> > > >>> processor-nodes, since those are the ones that would benefit most
> > from
> > > >>> the staleness metrics.
> > > >>>
> > > >>> Best,
> > > >>> Bruno
> > > >>>
> > > >>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > > sophie@confluent.io>
> > > >>> wrote:
> > > >>>>
> > > >>>> Yeah, the specific reason was just to align with the current
> > metrics.
> > > >>>>
> > > >>>> Is it better to conform than to be right? History has a lot to say
> > on
> > > >>> that
> > > >>>> matter
> > > >>>> but I'm not sure how much of it applies to the fine details of
> > metrics
> > > >>>> naming :P
> > > >>>>
> > > >>>> More seriously, I figured if people are looking at this metric
> > they're
> > > >>>> likely to
> > > >>>> be looking at all the others. Then naming this one "-mean" would
> > > probably
> > > >>>> lead some to conclude that the "-avg" suffix in the other metrics
> > has
> > > a
> > > >>>> different meaning.
> > > >>>>
> > > >>>> As for the percentiles, I actually like p99 (and p75) better. I'll
> > > swap
> > > >>>> that out
> > > >>>>
> > > >>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vvcephei@apache.org
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Thanks Sophie,
> > > >>>>>
> > > >>>>> I hope this isn't too nit-picky, but is there a reason to choose
> > > "avg"
> > > >>>>> instead
> > > >>>>> of "mean"? Maybe this is too paranoid, and I might be
> oversensitive
> > > >>> because
> > > >>>>> of the mistake I just made earlier, but it strikes me that "avg"
> is
> > > >>>>> actually
> > > >>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
> > is
> > > >>>>> specific.
> > > >>>>> I see other Kafka metrics with "avg", but none with "mean"; was
> > that
> > > >>> the
> > > >>>>> reason? If so, I'm +1.
> > > >>>>>
> > > >>>>> Regarding the names of the percentile, I actually couldn't find
> > _any_
> > > >>> other
> > > >>>>> metrics that use percentile. Was there a reason to choose "99th"
> as
> > > >>> opposed
> > > >>>>> to "p99" or any other scheme? This is not a criticism, I'm just
> > > >>> primarily
> > > >>>>> asking
> > > >>>>> for consistency's sake.
> > > >>>>>
> > > >>>>> Thanks again,
> > > >>>>> -John
> > > >>>>>
> > > >>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > >>>>>> Alright, I can get behind adding the min metric for the sake of
> > > >>> pretty
> > > >>>>>> graphs
> > > >>>>>> (and trivial computation).
> > > >>>>>>
> > > >>>>>> I'm still on the fence regarding the mean (or 50th percentile)
> > but I
> > > >>> can
> > > >>>>> see
> > > >>>>>> how users might expect it and find it a bit disorienting not to
> > > >>> have. So
> > > >>>>> the
> > > >>>>>> updated proposed metrics are
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>    - record-staleness-max [ms]
> > > >>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > > >>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > > >>>>>>    - record-staleness-avg [ms] *(mean)*
> > > >>>>>>    - record-staleness-min [ms]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <
> vvcephei@apache.org
> > >
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I
> guess
> > > >>> the
> > > >>>>> mean
> > > >>>>>>> seems more interesting to me than the median, but neither are
> as
> > > >>>>>>> interesting as the higher percentiles (99th and max).
> > > >>>>>>>
> > > >>>>>>> Min isn’t really important for any SLAs, but it does round out
> > the
> > > >>>>> mental
> > > >>>>>>> picture of the distribution. I’ve always graphed min along with
> > the
> > > >>>>> other
> > > >>>>>>> metrics to help me understand how fast the system can be, which
> > > >>> helps
> > > >>>>> in
> > > >>>>>>> optimization decisions. It’s also a relatively inexpensive
> metric
> > > >>> to
> > > >>>>>>> compute, so it might be nice to just throw it in.
> > > >>>>>>>
> > > >>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > >>>>>>>> G1:
> > > >>>>>>>> I was considering it as the "end-to-end latency *up* to the
> > > >>> specific
> > > >>>>>>> task"
> > > >>>>>>>> but
> > > >>>>>>>> I'm happy with "record-staleness" if that drives the point
> home
> > > >>>>> better.
> > > >>>>>>> So
> > > >>>>>>>> it's the
> > > >>>>>>>> "staleness of the record when it is received by that task" --
> > > >>> will
> > > >>>>> update
> > > >>>>>>>> the KIP
> > > >>>>>>>>
> > > >>>>>>>> B1/J:
> > > >>>>>>>> I'm struggling to imagine a case where the min would actually
> be
> > > >>>>> useful,
> > > >>>>>>>> rather than
> > > >>>>>>>> just intellectually interesting. I don't feel strongly that we
> > > >>>>> shouldn't
> > > >>>>>>>> add it, but that's
> > > >>>>>>>> why I didn't include it from the start. Can you enlighten me
> > > >>> with an
> > > >>>>>>>> example?
> > > >>>>>>>>
> > > >>>>>>>> I was also vaguely concerned about the overhead of adding
> > > >>> multiple
> > > >>>>>>>> percentile
> > > >>>>>>>> metrics. Do we have any data to indicate what kind of
> > performance
> > > >>>>> hit we
> > > >>>>>>>> take on
> > > >>>>>>>> metrics computation?
> > > >>>>>>>>
> > > >>>>>>>> Also, not to be too pedantic but the 50th percentile would be
> > the
> > > >>>>> median
> > > >>>>>>>> not the
> > > >>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > > >>> percentile, or
> > > >>>>>>> just
> > > >>>>>>>> one
> > > >>>>>>>> of the two?
> > > >>>>>>>>
> > > >>>>>>>> Thanks all!
> > > >>>>>>>> Sophie
> > > >>>>>>>>
> > > >>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > > >>> vvcephei@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > > >>>>>>>>>
> > > >>>>>>>>> Just some comments on the discussion so far:
> > > >>>>>>>>>
> > > >>>>>>>>> B2/G1:
> > > >>>>>>>>> In principle, it shouldn't matter whether we report "spans"
> or
> > > >>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
> > > >>> pretty
> > > >>>>>>>>> difficult to really measure (like time spent waiting in the
> > > >>>>> topics, or
> > > >>>>>>>>> time from the event happening to the ETL producer choosing to
> > > >>> send
> > > >>>>> it,
> > > >>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > > >>>>>>>>>
> > > >>>>>>>>> In other words, it's practically easier to compute spans by
> > > >>>>> subtracting
> > > >>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
> > > >>> spans.
> > > >>>>> You
> > > >>>>>>>>> can even just consider that the span computation from e2e
> > > >>> always
> > > >>>>> just
> > > >>>>>>>>> involves subtracting two numbers, whereas computing e2e
> latency
> > > >>>>> from
> > > >>>>>>>>> spans involves adding _all_ the spans leading up to the end
> you
> > > >>>>> care
> > > >>>>>>> about.
> > > >>>>>>>>>
> > > >>>>>>>>> It seems like people really prefer to have spans when they
> are
> > > >>>>>>> debugging
> > > >>>>>>>>> latency problems, whereas e2e latency is a more general
> > > >>> measurement
> > > >>>>>>>>> that basically every person/application cares about and
> should
> > > >>> be
> > > >>>>>>>>> monitoring.
> > > >>>>>>>>>
> > > >>>>>>>>> Altogether, it really seem to provide more value to more
> > > >>> people if
> > > >>>>> we
> > > >>>>>>>>> report
> > > >>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I
> think
> > > >>> I
> > > >>>>> have
> > > >>>>>>> no
> > > >>>>>>>>> preference, I'd defer to other peoples' intuition.
> > > >>>>>>>>>
> > > >>>>>>>>> G2:
> > > >>>>>>>>> I think the processor-node metric is nice, since the inside
> of
> > > >>> a
> > > >>>>> task
> > > >>>>>>> can
> > > >>>>>>>>> introduce a significant amount of latency in some cases.
> Plus,
> > > >>>>> it's a
> > > >>>>>>> more
> > > >>>>>>>>> direct measurement, if you really wanted to know (for the
> > > >>> purposes
> > > >>>>> of
> > > >>>>>>> IQ
> > > >>>>>>>>> or something) how long it takes source events to "show up" at
> > > >>> the
> > > >>>>>>> store.
> > > >>>>>>>>>
> > > >>>>>>>>> I think actually recording it at every processor could be
> > > >>>>> expensive,
> > > >>>>>>> but we
> > > >>>>>>>>> already record a bunch of metrics at the node level.
> > > >>>>>>>>>
> > > >>>>>>>>> B1:
> > > >>>>>>>>> I think 50% could be reasonable to record also. Even if it's
> a
> > > >>> poor
> > > >>>>>>> metric
> > > >>>>>>>>> for operational purposes, a lot of people might expect to see
> > > >>>>> "mean".
> > > >>>>>>>>> Actually,
> > > >>>>>>>>> I was surprised not to see "min". Is there a reason to leave
> it
> > > >>>>> off?
> > > >>>>>>>>>
> > > >>>>>>>>> I might suggest:
> > > >>>>>>>>> min, mean (50th), 75th, 99th, max
> > > >>>>>>>>>
> > > >>>>>>>>> B3:
> > > >>>>>>>>> I agree we should include late records (though not the ones
> we
> > > >>>>> drop).
> > > >>>>>>>>> It may be spiky, but only when there are legitimately some
> > > >>> records
> > > >>>>>>> with a
> > > >>>>>>>>> high end-to-end latency, which is the whole point of these
> > > >>> metrics.
> > > >>>>>>>>>
> > > >>>>>>>>> That's it! I don't think I have any other feedback, other
> than
> > > >>> a
> > > >>>>>>> request to
> > > >>>>>>>>> also report "min".
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> -John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > >>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1) The end-to-end latency includes both the processing
> > > >>> latency
> > > >>>>> of the
> > > >>>>>>>>> task
> > > >>>>>>>>>> and the latency spent sitting in intermediate topics. I have
> > > >>> a
> > > >>>>>>> similar
> > > >>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
> > > >>> a
> > > >>>>> task A
> > > >>>>>>>>>> actually measures the latency of the sub-topology up-to but
> > > >>> not
> > > >>>>>>> including
> > > >>>>>>>>>> the processing of A, which is a bit weird.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
> > > >>> since
> > > >>>>>>> today we
> > > >>>>>>>>>> already have several "latency" metrics already which are
> > > >>>>> measuring
> > > >>>>>>>>> elapsed
> > > >>>>>>>>>> system-time for processing a record / etc, while here we are
> > > >>>>>>> comparing
> > > >>>>>>>>> the
> > > >>>>>>>>>> system wallclock time with the record timestamp.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> > > >>> (note we
> > > >>>>>>> already
> > > >>>>>>>>>> have a "record-lateness" metric), in which case recording at
> > > >>> the
> > > >>>>>>>>>> system-time before we start processing the record sounds
> more
> > > >>>>>>> natural.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2) With that in mind, I'm wondering if the
> > > >>> processor-node-level
> > > >>>>> DEBUG
> > > >>>>>>>>>> metric is worth to add, given that we already have a
> > > >>> task-level
> > > >>>>>>>>> processing
> > > >>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
> > > >>>>> similar
> > > >>>>>>> to
> > > >>>>>>>>> the
> > > >>>>>>>>>> task-level e2e latency + task-level processing latency.
> > > >>>>> Personally I
> > > >>>>>>>>> think
> > > >>>>>>>>>> having a task-level record-staleness metric is sufficient.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Guozhang
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > >>>>>>>>> sophie@confluent.io>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
> > > >>> this
> > > >>>>>>>>> specific
> > > >>>>>>>>>>> metric, as
> > > >>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
> > > >>> the
> > > >>>>>>> latency
> > > >>>>>>>>> you
> > > >>>>>>>>>>> can
> > > >>>>>>>>>>> reasonably expect to see. I chose percentiles that would
> > > >>>>> hopefully
> > > >>>>>>>>> give a
> > > >>>>>>>>>>> good
> > > >>>>>>>>>>> sense of what *most* records will experience, and what
> > > >>> *close
> > > >>>>> to
> > > >>>>>>> all*
> > > >>>>>>>>>>> records
> > > >>>>>>>>>>> will.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> However I'm not married to these specific numbers and
> > > >>> could be
> > > >>>>>>>>> convinced.
> > > >>>>>>>>>>> Would be especially interested in hearing from users on
> > > >>> this.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
> > > >>>>> this KIP
> > > >>>>>>>>> since
> > > >>>>>>>>>>> users
> > > >>>>>>>>>>> can always compute it themselves by subtracting the
> > > >>> previous
> > > >>>>> node's
> > > >>>>>>>>>>> end-to-end latency. I guess we could do it either way since
> > > >>>>> you can
> > > >>>>>>>>> always
> > > >>>>>>>>>>> compute one from the other, but I think the end-to-end
> > > >>> latency
> > > >>>>>>> feels
> > > >>>>>>>>> more
> > > >>>>>>>>>>> valuable as it's main motivation is not to debug
> > > >>> bottlenecks
> > > >>>>> in the
> > > >>>>>>>>>>> topology but
> > > >>>>>>>>>>> to give users a sense of how long it takes arecord to be
> > > >>>>> reflected
> > > >>>>>>> in
> > > >>>>>>>>>>> certain parts
> > > >>>>>>>>>>> of the topology. For example this might be useful for users
> > > >>>>> who are
> > > >>>>>>>>>>> wondering
> > > >>>>>>>>>>> roughly when a record that was just produced will be
> > > >>> included
> > > >>>>> in
> > > >>>>>>> their
> > > >>>>>>>>> IQ
> > > >>>>>>>>>>> results.
> > > >>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
> > > >>> make
> > > >>>>>>> that
> > > >>>>>>>>> clear
> > > >>>>>>>>>>> enough
> > > >>>>>>>>>>> in the KIP's motivation.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> > > >>> short
> > > >>>>>>> answer is
> > > >>>>>>>>>>> "yes",
> > > >>>>>>>>>>> we will include late records. I added a paragraph to the
> > > >>> end
> > > >>>>> of the
> > > >>>>>>>>>>> Proposed
> > > >>>>>>>>>>> Changes section explaining the reasoning here, please let
> > > >>> me
> > > >>>>> know
> > > >>>>>>> if
> > > >>>>>>>>> you
> > > >>>>>>>>>>> have
> > > >>>>>>>>>>> any concerns.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > > >>>>>>> "process-latency",
> > > >>>>>>>>> that
> > > >>>>>>>>>>> metric
> > > >>>>>>>>>>> reflects the time for the literal Node#process method to
> > > >>> run
> > > >>>>>>> whereas
> > > >>>>>>>>> this
> > > >>>>>>>>>>> metric
> > > >>>>>>>>>>> would always be measured relative to the event timestamp.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That said, the naming collision there is pretty confusing
> > > >>> so
> > > >>>>> I've
> > > >>>>>>>>> renamed
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
> > > >>> better
> > > >>>>>>>>> reflects
> > > >>>>>>>>>>> the nature
> > > >>>>>>>>>>> of the metric anyway.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for the feedback!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > >>>>>>>>> reluctanthero104@gmail.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> > > >>>>> important
> > > >>>>>>> for
> > > >>>>>>>>>>>> understanding the bottleneck of the application.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> A couple of questions and ideas:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
> > > >>> max
> > > >>>>>>>>> percentiles?
> > > >>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> > > >>> well
> > > >>>>> in
> > > >>>>>>>>> production
> > > >>>>>>>>>>>> systems.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
> > > >>> if a
> > > >>>>>>> record
> > > >>>>>>>>> goes
> > > >>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > > >>> T(B->C) +
> > > >>>>>>> T(A->B) +
> > > >>>>>>>>>>> T(A)
> > > >>>>>>>>>>>> and so on, where P() represents the captured latency,
> > > >>> and T()
> > > >>>>>>>>> represents
> > > >>>>>>>>>>>> the time for transiting the records between two nodes,
> > > >>>>> including
> > > >>>>>>>>>>> processing
> > > >>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> > > >>>>> T(A->B)
> > > >>>>>>> are
> > > >>>>>>>>> more
> > > >>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> > > >>> there
> > > >>>>> is a
> > > >>>>>>>>> spike in
> > > >>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> > > >>> time.
> > > >>>>> In the
> > > >>>>>>>>> same
> > > >>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> > > >>> record
> > > >>>>> exits
> > > >>>>>>>>> from the
> > > >>>>>>>>>>>> sink as this marks the whole time this record spent
> > > >>> inside
> > > >>>>> the
> > > >>>>>>>>> funnel. Do
> > > >>>>>>>>>>>> you think we could have separate treatment for sink
> > > >>> nodes and
> > > >>>>>>> other
> > > >>>>>>>>>>>> nodes, so that other nodes only count the time receiving
> > > >>> the
> > > >>>>>>> record
> > > >>>>>>>>> from
> > > >>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
> > > >>>>> discuss
> > > >>>>>>>>> this
> > > >>>>>>>>>>>> alternative to see if it is reasonable.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> > > >>> well,
> > > >>>>> they
> > > >>>>>>>>> would
> > > >>>>>>>>>>>> create some really spiky graphs when the out-of-order
> > > >>>>> records are
> > > >>>>>>>>>>>> interleaving with on time records. Should we also supply
> > > >>> a
> > > >>>>> smooth
> > > >>>>>>>>> version
> > > >>>>>>>>>>>> of the latency metrics, or user should just take care of
> > > >>> it
> > > >>>>> by
> > > >>>>>>>>> themself?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
> > > >>>>> relation
> > > >>>>>>>>> with our
> > > >>>>>>>>>>>> existing processing latency metrics, could you add some
> > > >>>>> context
> > > >>>>>>> on
> > > >>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
> > > >>> the
> > > >>>>>>> best?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Boyang
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > >>>>>>>>> sophie@confluent.io
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hey all,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> > > >>> to
> > > >>>>> add
> > > >>>>>>>>> end-to-end
> > > >>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>> Sophie
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>> -- Guozhang
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
@Matthias
Regarding tracking the 50th percentile, I'll refer you to the 4:53 mark of
the video
*you* linked: https://youtu.be/lJ8ydIuPFeU?t=293

And that's what he says about the 95th percentile! Imagine what he would
say about
the 50th :P

But seriously, since we can't seem to agree that the mean or 50th
percentile is actually
useful I'm inclined to resurrect my original proposal, neither. But I think
that's a good
argument against the 75th, which I admittedly chose somewhat arbitrarily as
an
intermediate between the 50th and the higher percentiles. How about:

-max
-p99
-p90
-min

with p50/mean still up for debate if anyone feels strongly for either of
them.

Regarding the name, I've already flip-flopped on this so I'm definitely
still open to
further arguments. But the reason for changing it from end-to-end-latency
(which
is similar to what you propose) is that this metric technically reflects
how old (ie how "stale")
the record is when it's *received* by the operator, not when it's processed
by the operator.
It seemed like there was the potential for confusion that
"end-to-end-latency" might
represent the latency from the event creation to the time the processor is
done
processing it.

@John
I'd rather err on the side of "not-enough" metrics as we can always add
this to the
stateless metrics later on. If we decide to measure the time at every node
and don't
find any evidence of a serious performance impact, and users indicate they
would
like to see this metric at all nodes, then we can easily start reporting
them as well.
WDYT?

That said, sink nodes seem like a reasonable exception to the rule.
Obviously users
should be able to detect the time when the record reaches the output topic
but that
still leaves a gap in understanding how long the production latency was.
This mirrors
the consumption latency that is exposed by the task-level metrics, which
are measured
at the source node. For good symmetry what if we actually expose both the
source
and sink latency at the task-level? ie report both sets of statistical
measurements with
the additional tag -source/-sink

@Bill
Thanks for the comment about regarding the min! I hadn't considered that
and it's
quite useful to think about how and what is useful from a users point of
view.

Regarding your second. point, I'm inclined to leave that as an
implementation detail
but my take would be that the user should be allowed to control the record
timestamp
used for this with the timestamp extractor. My impression is that users may
often embed
the actual event time in the payload for whatever reason, and this
represents the "true"
timestamp as far as the Streams topology is concerned.


On Fri, May 15, 2020 at 11:05 AM Bill Bejeck <bb...@gmail.com> wrote:

> Thanks for the KIP, Sophie, this will be a useful metric to add.
>
> Regarding tracking min, I  think it could be valuable for users to discern
> which part of their topologies are more efficient since this is a
> task-level metric.  I realize everyone seems to be on board with including
> min anyway, but I wanted to add my 2 cents on this topic should we decide
> to revisit adding min or not.
>
> I do have a question regarding the calculation of staleness.
> Is there going to be a consideration for timestamp extractors? Users could
> prefer to use a timestamp embedded in the payload, and it could skew the
> measurements.
> I was wondering if we should specify in the KIP if setting the arrival time
> is always going to come from the record timestamp, or is this an
> implementation detail we can cover in the PR?
>
> Thanks!
> Bill
>
> On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > Thanks for the KIP Sophie.
> >
> > I think it's not useful to record the avg/mean; it sensitive to
> > outliers. We should rather track the median (50th percentile).
> >
> > Not sure if tracking min is useful, but I am also ok to track it.
> >
> > However, I find it odd to track 75th percentile. Standard measures would
> > the 90th or 95th -- I guess we don't need both, so maybe picking 90th
> > might be more useful?
> >
> > About the name: "staleness" wound really odd, and if fact the metric
> > does capture "latency" so we should call it "latency". I understand the
> > issue that we already have a latency metric. So maybe we could call it
> > `record-e2e-latency-*` ?
> >
> > While I agree that we should include out-or-order data (the KIP should
> > talk about `out-of-order` data, not `late` data; data is only `late` if
> > it's out-of-order and if it's dropped), I don't really understand why
> > the new metric would help to configure grace period or retention time?
> > As you mention in the KIP, both are define as max difference of
> > `event-time - stream-time` and thus the new metric that takes
> > system-/wallclock-time into account does not seem to help at all.
> >
> >
> > Btw: there is a great talk about "How NOT to Measure Latency" by Gil
> > Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
> >
> >
> > -Matthias
> >
> >
> > On 5/14/20 7:17 PM, John Roesler wrote:
> > > Hi Sophie,
> > >
> > > It seems like there would still be plenty of use cases for recording
> > > this metric at all processors and not just stateful ones, but I'm happy
> > > to suspend my arguments for now. Since you're proposing to keep
> > > them at the processor-node level, it will be seamless later to add
> > > in the stateless processors if we want. As a wise man once said,
> > > "Adding is always easier than removing."
> > >
> > > Regarding the time measurement, it's an implementation detail
> > > we don't need to consider in the KIP. Nevertheless, I'd greatly
> > > prefer to measure the system time again when recording the
> > > metric. I don't think we've seen any evidence that proves this
> > > would harm performance, and the amount of inaccuracy using
> > > the cached system time could incur is actually substantial. But,
> > > if you want to just "not mention this" in the KIP, we can defer to
> > > the actual PR discussion, at which time we're in a better position
> > > to use benchmarks, etc., to make the call.
> > >
> > > Along the lines of the measurement accuracy discussion, one
> > > minor thought I had is that maybe we should consider measuring
> > > the task staleness metric at the sink, rather than the source, so that
> > > it includes the processing latency of the task itself, not just the
> > latency
> > > of everything up to, but not including, the task (which seems confusing
> > > for users). I guess this could also be an implementation detail,
> though.
> > >
> > > Thanks for the update,
> > > -John
> > >
> > > On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> > >> Hey all,
> > >>
> > >> After discussing with Bruno I'd like to propose a small amendment,
> > >> which is to record the processor-node-level metrics only for
> *stateful*
> > >> *operators*. They would still be considered a "processor-node-level"
> > >> metric and not a "state-store-level" metric as the staleness is still
> > >> a property of the node rather than of the state itself. However, it
> > seems
> > >> that this information is primarily useful for stateful operators that
> > might
> > >> be exposing state via IQ or otherwise dependent on the record time
> > >> unlike a stateless operator.
> > >>
> > >> It's worth calling out that recent performance improvements to the
> > metrics
> > >> framework mean that we no longer fetch the system time at the operator
> > >> level, but only once per task. In other words the system time is not
> > updated
> > >> between each process as a record flows through the subtopology, so
> > >> debugging the processor-level latency via the stateleness will not be
> > >> possible.Note that this doesn't mean the operator-level metrics are
> not
> > >> *useful* relative to the task-level metric. Upstream caching and/or
> > >> suppression
> > >> can still cause a record's staleness at some downstream stateful
> > operator
> > >> to deviate from the task-level staleness (recorded at the source
> node).
> > >>
> > >> Please let me know if you have any concerns about this change. The
> > >> KIP has been updated with the new proposal
> > >>
> > >> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
> > wrote:
> > >>
> > >>> Hi Sophie,
> > >>>
> > >>> Thank you for the KIP.
> > >>>
> > >>> The KIP looks good to me.
> > >>>
> > >>> 50th percentile:
> > >>> I think we do not need it now. If we need it, we can add it. Here the
> > >>> old truism applies: Adding is always easier than removing.
> > >>>
> > >>> processor-node-level metrics:
> > >>> I think it is good to have the staleness metrics also on
> > >>> processor-node-level. If we do not want to record them on all
> > >>> processor nodes, you could restrict the recording to stateful
> > >>> processor-nodes, since those are the ones that would benefit most
> from
> > >>> the staleness metrics.
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> > sophie@confluent.io>
> > >>> wrote:
> > >>>>
> > >>>> Yeah, the specific reason was just to align with the current
> metrics.
> > >>>>
> > >>>> Is it better to conform than to be right? History has a lot to say
> on
> > >>> that
> > >>>> matter
> > >>>> but I'm not sure how much of it applies to the fine details of
> metrics
> > >>>> naming :P
> > >>>>
> > >>>> More seriously, I figured if people are looking at this metric
> they're
> > >>>> likely to
> > >>>> be looking at all the others. Then naming this one "-mean" would
> > probably
> > >>>> lead some to conclude that the "-avg" suffix in the other metrics
> has
> > a
> > >>>> different meaning.
> > >>>>
> > >>>> As for the percentiles, I actually like p99 (and p75) better. I'll
> > swap
> > >>>> that out
> > >>>>
> > >>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> Thanks Sophie,
> > >>>>>
> > >>>>> I hope this isn't too nit-picky, but is there a reason to choose
> > "avg"
> > >>>>> instead
> > >>>>> of "mean"? Maybe this is too paranoid, and I might be oversensitive
> > >>> because
> > >>>>> of the mistake I just made earlier, but it strikes me that "avg" is
> > >>>>> actually
> > >>>>> ambiguous, as it refers to a family of statistics, whereas "mean"
> is
> > >>>>> specific.
> > >>>>> I see other Kafka metrics with "avg", but none with "mean"; was
> that
> > >>> the
> > >>>>> reason? If so, I'm +1.
> > >>>>>
> > >>>>> Regarding the names of the percentile, I actually couldn't find
> _any_
> > >>> other
> > >>>>> metrics that use percentile. Was there a reason to choose "99th" as
> > >>> opposed
> > >>>>> to "p99" or any other scheme? This is not a criticism, I'm just
> > >>> primarily
> > >>>>> asking
> > >>>>> for consistency's sake.
> > >>>>>
> > >>>>> Thanks again,
> > >>>>> -John
> > >>>>>
> > >>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > >>>>>> Alright, I can get behind adding the min metric for the sake of
> > >>> pretty
> > >>>>>> graphs
> > >>>>>> (and trivial computation).
> > >>>>>>
> > >>>>>> I'm still on the fence regarding the mean (or 50th percentile)
> but I
> > >>> can
> > >>>>> see
> > >>>>>> how users might expect it and find it a bit disorienting not to
> > >>> have. So
> > >>>>> the
> > >>>>>> updated proposed metrics are
> > >>>>>>
> > >>>>>>
> > >>>>>>    - record-staleness-max [ms]
> > >>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> > >>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> > >>>>>>    - record-staleness-avg [ms] *(mean)*
> > >>>>>>    - record-staleness-min [ms]
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <vvcephei@apache.org
> >
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I guess
> > >>> the
> > >>>>> mean
> > >>>>>>> seems more interesting to me than the median, but neither are as
> > >>>>>>> interesting as the higher percentiles (99th and max).
> > >>>>>>>
> > >>>>>>> Min isn’t really important for any SLAs, but it does round out
> the
> > >>>>> mental
> > >>>>>>> picture of the distribution. I’ve always graphed min along with
> the
> > >>>>> other
> > >>>>>>> metrics to help me understand how fast the system can be, which
> > >>> helps
> > >>>>> in
> > >>>>>>> optimization decisions. It’s also a relatively inexpensive metric
> > >>> to
> > >>>>>>> compute, so it might be nice to just throw it in.
> > >>>>>>>
> > >>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > >>>>>>>> G1:
> > >>>>>>>> I was considering it as the "end-to-end latency *up* to the
> > >>> specific
> > >>>>>>> task"
> > >>>>>>>> but
> > >>>>>>>> I'm happy with "record-staleness" if that drives the point home
> > >>>>> better.
> > >>>>>>> So
> > >>>>>>>> it's the
> > >>>>>>>> "staleness of the record when it is received by that task" --
> > >>> will
> > >>>>> update
> > >>>>>>>> the KIP
> > >>>>>>>>
> > >>>>>>>> B1/J:
> > >>>>>>>> I'm struggling to imagine a case where the min would actually be
> > >>>>> useful,
> > >>>>>>>> rather than
> > >>>>>>>> just intellectually interesting. I don't feel strongly that we
> > >>>>> shouldn't
> > >>>>>>>> add it, but that's
> > >>>>>>>> why I didn't include it from the start. Can you enlighten me
> > >>> with an
> > >>>>>>>> example?
> > >>>>>>>>
> > >>>>>>>> I was also vaguely concerned about the overhead of adding
> > >>> multiple
> > >>>>>>>> percentile
> > >>>>>>>> metrics. Do we have any data to indicate what kind of
> performance
> > >>>>> hit we
> > >>>>>>>> take on
> > >>>>>>>> metrics computation?
> > >>>>>>>>
> > >>>>>>>> Also, not to be too pedantic but the 50th percentile would be
> the
> > >>>>> median
> > >>>>>>>> not the
> > >>>>>>>> mean. Would you propose to add the mean *and* the 50th
> > >>> percentile, or
> > >>>>>>> just
> > >>>>>>>> one
> > >>>>>>>> of the two?
> > >>>>>>>>
> > >>>>>>>> Thanks all!
> > >>>>>>>> Sophie
> > >>>>>>>>
> > >>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > >>> vvcephei@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> > >>>>>>>>>
> > >>>>>>>>> Just some comments on the discussion so far:
> > >>>>>>>>>
> > >>>>>>>>> B2/G1:
> > >>>>>>>>> In principle, it shouldn't matter whether we report "spans" or
> > >>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
> > >>> pretty
> > >>>>>>>>> difficult to really measure (like time spent waiting in the
> > >>>>> topics, or
> > >>>>>>>>> time from the event happening to the ETL producer choosing to
> > >>> send
> > >>>>> it,
> > >>>>>>>>> or time spent in send/receive buffers, etc., etc.
> > >>>>>>>>>
> > >>>>>>>>> In other words, it's practically easier to compute spans by
> > >>>>> subtracting
> > >>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
> > >>> spans.
> > >>>>> You
> > >>>>>>>>> can even just consider that the span computation from e2e
> > >>> always
> > >>>>> just
> > >>>>>>>>> involves subtracting two numbers, whereas computing e2e latency
> > >>>>> from
> > >>>>>>>>> spans involves adding _all_ the spans leading up to the end you
> > >>>>> care
> > >>>>>>> about.
> > >>>>>>>>>
> > >>>>>>>>> It seems like people really prefer to have spans when they are
> > >>>>>>> debugging
> > >>>>>>>>> latency problems, whereas e2e latency is a more general
> > >>> measurement
> > >>>>>>>>> that basically every person/application cares about and should
> > >>> be
> > >>>>>>>>> monitoring.
> > >>>>>>>>>
> > >>>>>>>>> Altogether, it really seem to provide more value to more
> > >>> people if
> > >>>>> we
> > >>>>>>>>> report
> > >>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I think
> > >>> I
> > >>>>> have
> > >>>>>>> no
> > >>>>>>>>> preference, I'd defer to other peoples' intuition.
> > >>>>>>>>>
> > >>>>>>>>> G2:
> > >>>>>>>>> I think the processor-node metric is nice, since the inside of
> > >>> a
> > >>>>> task
> > >>>>>>> can
> > >>>>>>>>> introduce a significant amount of latency in some cases. Plus,
> > >>>>> it's a
> > >>>>>>> more
> > >>>>>>>>> direct measurement, if you really wanted to know (for the
> > >>> purposes
> > >>>>> of
> > >>>>>>> IQ
> > >>>>>>>>> or something) how long it takes source events to "show up" at
> > >>> the
> > >>>>>>> store.
> > >>>>>>>>>
> > >>>>>>>>> I think actually recording it at every processor could be
> > >>>>> expensive,
> > >>>>>>> but we
> > >>>>>>>>> already record a bunch of metrics at the node level.
> > >>>>>>>>>
> > >>>>>>>>> B1:
> > >>>>>>>>> I think 50% could be reasonable to record also. Even if it's a
> > >>> poor
> > >>>>>>> metric
> > >>>>>>>>> for operational purposes, a lot of people might expect to see
> > >>>>> "mean".
> > >>>>>>>>> Actually,
> > >>>>>>>>> I was surprised not to see "min". Is there a reason to leave it
> > >>>>> off?
> > >>>>>>>>>
> > >>>>>>>>> I might suggest:
> > >>>>>>>>> min, mean (50th), 75th, 99th, max
> > >>>>>>>>>
> > >>>>>>>>> B3:
> > >>>>>>>>> I agree we should include late records (though not the ones we
> > >>>>> drop).
> > >>>>>>>>> It may be spiky, but only when there are legitimately some
> > >>> records
> > >>>>>>> with a
> > >>>>>>>>> high end-to-end latency, which is the whole point of these
> > >>> metrics.
> > >>>>>>>>>
> > >>>>>>>>> That's it! I don't think I have any other feedback, other than
> > >>> a
> > >>>>>>> request to
> > >>>>>>>>> also report "min".
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> -John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > >>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> > >>>>>>>>>>
> > >>>>>>>>>> 1) The end-to-end latency includes both the processing
> > >>> latency
> > >>>>> of the
> > >>>>>>>>> task
> > >>>>>>>>>> and the latency spent sitting in intermediate topics. I have
> > >>> a
> > >>>>>>> similar
> > >>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
> > >>> a
> > >>>>> task A
> > >>>>>>>>>> actually measures the latency of the sub-topology up-to but
> > >>> not
> > >>>>>>> including
> > >>>>>>>>>> the processing of A, which is a bit weird.
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
> > >>> since
> > >>>>>>> today we
> > >>>>>>>>>> already have several "latency" metrics already which are
> > >>>>> measuring
> > >>>>>>>>> elapsed
> > >>>>>>>>>> system-time for processing a record / etc, while here we are
> > >>>>>>> comparing
> > >>>>>>>>> the
> > >>>>>>>>>> system wallclock time with the record timestamp.
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> > >>> (note we
> > >>>>>>> already
> > >>>>>>>>>> have a "record-lateness" metric), in which case recording at
> > >>> the
> > >>>>>>>>>> system-time before we start processing the record sounds more
> > >>>>>>> natural.
> > >>>>>>>>>>
> > >>>>>>>>>> 2) With that in mind, I'm wondering if the
> > >>> processor-node-level
> > >>>>> DEBUG
> > >>>>>>>>>> metric is worth to add, given that we already have a
> > >>> task-level
> > >>>>>>>>> processing
> > >>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
> > >>>>> similar
> > >>>>>>> to
> > >>>>>>>>> the
> > >>>>>>>>>> task-level e2e latency + task-level processing latency.
> > >>>>> Personally I
> > >>>>>>>>> think
> > >>>>>>>>>> having a task-level record-staleness metric is sufficient.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Guozhang
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > >>>>>>>>> sophie@confluent.io>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
> > >>> this
> > >>>>>>>>> specific
> > >>>>>>>>>>> metric, as
> > >>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
> > >>> the
> > >>>>>>> latency
> > >>>>>>>>> you
> > >>>>>>>>>>> can
> > >>>>>>>>>>> reasonably expect to see. I chose percentiles that would
> > >>>>> hopefully
> > >>>>>>>>> give a
> > >>>>>>>>>>> good
> > >>>>>>>>>>> sense of what *most* records will experience, and what
> > >>> *close
> > >>>>> to
> > >>>>>>> all*
> > >>>>>>>>>>> records
> > >>>>>>>>>>> will.
> > >>>>>>>>>>>
> > >>>>>>>>>>> However I'm not married to these specific numbers and
> > >>> could be
> > >>>>>>>>> convinced.
> > >>>>>>>>>>> Would be especially interested in hearing from users on
> > >>> this.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
> > >>>>> this KIP
> > >>>>>>>>> since
> > >>>>>>>>>>> users
> > >>>>>>>>>>> can always compute it themselves by subtracting the
> > >>> previous
> > >>>>> node's
> > >>>>>>>>>>> end-to-end latency. I guess we could do it either way since
> > >>>>> you can
> > >>>>>>>>> always
> > >>>>>>>>>>> compute one from the other, but I think the end-to-end
> > >>> latency
> > >>>>>>> feels
> > >>>>>>>>> more
> > >>>>>>>>>>> valuable as it's main motivation is not to debug
> > >>> bottlenecks
> > >>>>> in the
> > >>>>>>>>>>> topology but
> > >>>>>>>>>>> to give users a sense of how long it takes arecord to be
> > >>>>> reflected
> > >>>>>>> in
> > >>>>>>>>>>> certain parts
> > >>>>>>>>>>> of the topology. For example this might be useful for users
> > >>>>> who are
> > >>>>>>>>>>> wondering
> > >>>>>>>>>>> roughly when a record that was just produced will be
> > >>> included
> > >>>>> in
> > >>>>>>> their
> > >>>>>>>>> IQ
> > >>>>>>>>>>> results.
> > >>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
> > >>> make
> > >>>>>>> that
> > >>>>>>>>> clear
> > >>>>>>>>>>> enough
> > >>>>>>>>>>> in the KIP's motivation.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> > >>> short
> > >>>>>>> answer is
> > >>>>>>>>>>> "yes",
> > >>>>>>>>>>> we will include late records. I added a paragraph to the
> > >>> end
> > >>>>> of the
> > >>>>>>>>>>> Proposed
> > >>>>>>>>>>> Changes section explaining the reasoning here, please let
> > >>> me
> > >>>>> know
> > >>>>>>> if
> > >>>>>>>>> you
> > >>>>>>>>>>> have
> > >>>>>>>>>>> any concerns.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 4. Assuming you're referring to the existing metric
> > >>>>>>> "process-latency",
> > >>>>>>>>> that
> > >>>>>>>>>>> metric
> > >>>>>>>>>>> reflects the time for the literal Node#process method to
> > >>> run
> > >>>>>>> whereas
> > >>>>>>>>> this
> > >>>>>>>>>>> metric
> > >>>>>>>>>>> would always be measured relative to the event timestamp.
> > >>>>>>>>>>>
> > >>>>>>>>>>> That said, the naming collision there is pretty confusing
> > >>> so
> > >>>>> I've
> > >>>>>>>>> renamed
> > >>>>>>>>>>> the
> > >>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
> > >>> better
> > >>>>>>>>> reflects
> > >>>>>>>>>>> the nature
> > >>>>>>>>>>> of the metric anyway.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the feedback!
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > >>>>>>>>> reluctanthero104@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> > >>>>> important
> > >>>>>>> for
> > >>>>>>>>>>>> understanding the bottleneck of the application.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> A couple of questions and ideas:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
> > >>> max
> > >>>>>>>>> percentiles?
> > >>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> > >>> well
> > >>>>> in
> > >>>>>>>>> production
> > >>>>>>>>>>>> systems.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
> > >>> if a
> > >>>>>>> record
> > >>>>>>>>> goes
> > >>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > >>> T(B->C) +
> > >>>>>>> T(A->B) +
> > >>>>>>>>>>> T(A)
> > >>>>>>>>>>>> and so on, where P() represents the captured latency,
> > >>> and T()
> > >>>>>>>>> represents
> > >>>>>>>>>>>> the time for transiting the records between two nodes,
> > >>>>> including
> > >>>>>>>>>>> processing
> > >>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> > >>>>> T(A->B)
> > >>>>>>> are
> > >>>>>>>>> more
> > >>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> > >>> there
> > >>>>> is a
> > >>>>>>>>> spike in
> > >>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> > >>> time.
> > >>>>> In the
> > >>>>>>>>> same
> > >>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> > >>> record
> > >>>>> exits
> > >>>>>>>>> from the
> > >>>>>>>>>>>> sink as this marks the whole time this record spent
> > >>> inside
> > >>>>> the
> > >>>>>>>>> funnel. Do
> > >>>>>>>>>>>> you think we could have separate treatment for sink
> > >>> nodes and
> > >>>>>>> other
> > >>>>>>>>>>>> nodes, so that other nodes only count the time receiving
> > >>> the
> > >>>>>>> record
> > >>>>>>>>> from
> > >>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
> > >>>>> discuss
> > >>>>>>>>> this
> > >>>>>>>>>>>> alternative to see if it is reasonable.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> > >>> well,
> > >>>>> they
> > >>>>>>>>> would
> > >>>>>>>>>>>> create some really spiky graphs when the out-of-order
> > >>>>> records are
> > >>>>>>>>>>>> interleaving with on time records. Should we also supply
> > >>> a
> > >>>>> smooth
> > >>>>>>>>> version
> > >>>>>>>>>>>> of the latency metrics, or user should just take care of
> > >>> it
> > >>>>> by
> > >>>>>>>>> themself?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
> > >>>>> relation
> > >>>>>>>>> with our
> > >>>>>>>>>>>> existing processing latency metrics, could you add some
> > >>>>> context
> > >>>>>>> on
> > >>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
> > >>> the
> > >>>>>>> best?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Boyang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > >>>>>>>>> sophie@confluent.io
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> > >>> to
> > >>>>> add
> > >>>>>>>>> end-to-end
> > >>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>> Sophie
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> -- Guozhang
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Bill Bejeck <bb...@gmail.com>.
Thanks for the KIP, Sophie, this will be a useful metric to add.

Regarding tracking min, I  think it could be valuable for users to discern
which part of their topologies are more efficient since this is a
task-level metric.  I realize everyone seems to be on board with including
min anyway, but I wanted to add my 2 cents on this topic should we decide
to revisit adding min or not.

I do have a question regarding the calculation of staleness.
Is there going to be a consideration for timestamp extractors? Users could
prefer to use a timestamp embedded in the payload, and it could skew the
measurements.
I was wondering if we should specify in the KIP if setting the arrival time
is always going to come from the record timestamp, or is this an
implementation detail we can cover in the PR?

Thanks!
Bill

On Fri, May 15, 2020 at 1:11 AM Matthias J. Sax <mj...@apache.org> wrote:

> Thanks for the KIP Sophie.
>
> I think it's not useful to record the avg/mean; it sensitive to
> outliers. We should rather track the median (50th percentile).
>
> Not sure if tracking min is useful, but I am also ok to track it.
>
> However, I find it odd to track 75th percentile. Standard measures would
> the 90th or 95th -- I guess we don't need both, so maybe picking 90th
> might be more useful?
>
> About the name: "staleness" wound really odd, and if fact the metric
> does capture "latency" so we should call it "latency". I understand the
> issue that we already have a latency metric. So maybe we could call it
> `record-e2e-latency-*` ?
>
> While I agree that we should include out-or-order data (the KIP should
> talk about `out-of-order` data, not `late` data; data is only `late` if
> it's out-of-order and if it's dropped), I don't really understand why
> the new metric would help to configure grace period or retention time?
> As you mention in the KIP, both are define as max difference of
> `event-time - stream-time` and thus the new metric that takes
> system-/wallclock-time into account does not seem to help at all.
>
>
> Btw: there is a great talk about "How NOT to Measure Latency" by Gil
> Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU
>
>
> -Matthias
>
>
> On 5/14/20 7:17 PM, John Roesler wrote:
> > Hi Sophie,
> >
> > It seems like there would still be plenty of use cases for recording
> > this metric at all processors and not just stateful ones, but I'm happy
> > to suspend my arguments for now. Since you're proposing to keep
> > them at the processor-node level, it will be seamless later to add
> > in the stateless processors if we want. As a wise man once said,
> > "Adding is always easier than removing."
> >
> > Regarding the time measurement, it's an implementation detail
> > we don't need to consider in the KIP. Nevertheless, I'd greatly
> > prefer to measure the system time again when recording the
> > metric. I don't think we've seen any evidence that proves this
> > would harm performance, and the amount of inaccuracy using
> > the cached system time could incur is actually substantial. But,
> > if you want to just "not mention this" in the KIP, we can defer to
> > the actual PR discussion, at which time we're in a better position
> > to use benchmarks, etc., to make the call.
> >
> > Along the lines of the measurement accuracy discussion, one
> > minor thought I had is that maybe we should consider measuring
> > the task staleness metric at the sink, rather than the source, so that
> > it includes the processing latency of the task itself, not just the
> latency
> > of everything up to, but not including, the task (which seems confusing
> > for users). I guess this could also be an implementation detail, though.
> >
> > Thanks for the update,
> > -John
> >
> > On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> >> Hey all,
> >>
> >> After discussing with Bruno I'd like to propose a small amendment,
> >> which is to record the processor-node-level metrics only for *stateful*
> >> *operators*. They would still be considered a "processor-node-level"
> >> metric and not a "state-store-level" metric as the staleness is still
> >> a property of the node rather than of the state itself. However, it
> seems
> >> that this information is primarily useful for stateful operators that
> might
> >> be exposing state via IQ or otherwise dependent on the record time
> >> unlike a stateless operator.
> >>
> >> It's worth calling out that recent performance improvements to the
> metrics
> >> framework mean that we no longer fetch the system time at the operator
> >> level, but only once per task. In other words the system time is not
> updated
> >> between each process as a record flows through the subtopology, so
> >> debugging the processor-level latency via the stateleness will not be
> >> possible.Note that this doesn't mean the operator-level metrics are not
> >> *useful* relative to the task-level metric. Upstream caching and/or
> >> suppression
> >> can still cause a record's staleness at some downstream stateful
> operator
> >> to deviate from the task-level staleness (recorded at the source node).
> >>
> >> Please let me know if you have any concerns about this change. The
> >> KIP has been updated with the new proposal
> >>
> >> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io>
> wrote:
> >>
> >>> Hi Sophie,
> >>>
> >>> Thank you for the KIP.
> >>>
> >>> The KIP looks good to me.
> >>>
> >>> 50th percentile:
> >>> I think we do not need it now. If we need it, we can add it. Here the
> >>> old truism applies: Adding is always easier than removing.
> >>>
> >>> processor-node-level metrics:
> >>> I think it is good to have the staleness metrics also on
> >>> processor-node-level. If we do not want to record them on all
> >>> processor nodes, you could restrict the recording to stateful
> >>> processor-nodes, since those are the ones that would benefit most from
> >>> the staleness metrics.
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <
> sophie@confluent.io>
> >>> wrote:
> >>>>
> >>>> Yeah, the specific reason was just to align with the current metrics.
> >>>>
> >>>> Is it better to conform than to be right? History has a lot to say on
> >>> that
> >>>> matter
> >>>> but I'm not sure how much of it applies to the fine details of metrics
> >>>> naming :P
> >>>>
> >>>> More seriously, I figured if people are looking at this metric they're
> >>>> likely to
> >>>> be looking at all the others. Then naming this one "-mean" would
> probably
> >>>> lead some to conclude that the "-avg" suffix in the other metrics has
> a
> >>>> different meaning.
> >>>>
> >>>> As for the percentiles, I actually like p99 (and p75) better. I'll
> swap
> >>>> that out
> >>>>
> >>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org>
> >>> wrote:
> >>>>
> >>>>> Thanks Sophie,
> >>>>>
> >>>>> I hope this isn't too nit-picky, but is there a reason to choose
> "avg"
> >>>>> instead
> >>>>> of "mean"? Maybe this is too paranoid, and I might be oversensitive
> >>> because
> >>>>> of the mistake I just made earlier, but it strikes me that "avg" is
> >>>>> actually
> >>>>> ambiguous, as it refers to a family of statistics, whereas "mean" is
> >>>>> specific.
> >>>>> I see other Kafka metrics with "avg", but none with "mean"; was that
> >>> the
> >>>>> reason? If so, I'm +1.
> >>>>>
> >>>>> Regarding the names of the percentile, I actually couldn't find _any_
> >>> other
> >>>>> metrics that use percentile. Was there a reason to choose "99th" as
> >>> opposed
> >>>>> to "p99" or any other scheme? This is not a criticism, I'm just
> >>> primarily
> >>>>> asking
> >>>>> for consistency's sake.
> >>>>>
> >>>>> Thanks again,
> >>>>> -John
> >>>>>
> >>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> >>>>>> Alright, I can get behind adding the min metric for the sake of
> >>> pretty
> >>>>>> graphs
> >>>>>> (and trivial computation).
> >>>>>>
> >>>>>> I'm still on the fence regarding the mean (or 50th percentile) but I
> >>> can
> >>>>> see
> >>>>>> how users might expect it and find it a bit disorienting not to
> >>> have. So
> >>>>> the
> >>>>>> updated proposed metrics are
> >>>>>>
> >>>>>>
> >>>>>>    - record-staleness-max [ms]
> >>>>>>    - record-staleness-99th [ms] *(99th percentile)*
> >>>>>>    - record-staleness-75th [ms] *(75th percentile)*
> >>>>>>    - record-staleness-avg [ms] *(mean)*
> >>>>>>    - record-staleness-min [ms]
> >>>>>>
> >>>>>>
> >>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I guess
> >>> the
> >>>>> mean
> >>>>>>> seems more interesting to me than the median, but neither are as
> >>>>>>> interesting as the higher percentiles (99th and max).
> >>>>>>>
> >>>>>>> Min isn’t really important for any SLAs, but it does round out the
> >>>>> mental
> >>>>>>> picture of the distribution. I’ve always graphed min along with the
> >>>>> other
> >>>>>>> metrics to help me understand how fast the system can be, which
> >>> helps
> >>>>> in
> >>>>>>> optimization decisions. It’s also a relatively inexpensive metric
> >>> to
> >>>>>>> compute, so it might be nice to just throw it in.
> >>>>>>>
> >>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> >>>>>>>> G1:
> >>>>>>>> I was considering it as the "end-to-end latency *up* to the
> >>> specific
> >>>>>>> task"
> >>>>>>>> but
> >>>>>>>> I'm happy with "record-staleness" if that drives the point home
> >>>>> better.
> >>>>>>> So
> >>>>>>>> it's the
> >>>>>>>> "staleness of the record when it is received by that task" --
> >>> will
> >>>>> update
> >>>>>>>> the KIP
> >>>>>>>>
> >>>>>>>> B1/J:
> >>>>>>>> I'm struggling to imagine a case where the min would actually be
> >>>>> useful,
> >>>>>>>> rather than
> >>>>>>>> just intellectually interesting. I don't feel strongly that we
> >>>>> shouldn't
> >>>>>>>> add it, but that's
> >>>>>>>> why I didn't include it from the start. Can you enlighten me
> >>> with an
> >>>>>>>> example?
> >>>>>>>>
> >>>>>>>> I was also vaguely concerned about the overhead of adding
> >>> multiple
> >>>>>>>> percentile
> >>>>>>>> metrics. Do we have any data to indicate what kind of performance
> >>>>> hit we
> >>>>>>>> take on
> >>>>>>>> metrics computation?
> >>>>>>>>
> >>>>>>>> Also, not to be too pedantic but the 50th percentile would be the
> >>>>> median
> >>>>>>>> not the
> >>>>>>>> mean. Would you propose to add the mean *and* the 50th
> >>> percentile, or
> >>>>>>> just
> >>>>>>>> one
> >>>>>>>> of the two?
> >>>>>>>>
> >>>>>>>> Thanks all!
> >>>>>>>> Sophie
> >>>>>>>>
> >>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
> >>> vvcephei@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hello all, and thanks for the KIP, Sophie,
> >>>>>>>>>
> >>>>>>>>> Just some comments on the discussion so far:
> >>>>>>>>>
> >>>>>>>>> B2/G1:
> >>>>>>>>> In principle, it shouldn't matter whether we report "spans" or
> >>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
> >>> pretty
> >>>>>>>>> difficult to really measure (like time spent waiting in the
> >>>>> topics, or
> >>>>>>>>> time from the event happening to the ETL producer choosing to
> >>> send
> >>>>> it,
> >>>>>>>>> or time spent in send/receive buffers, etc., etc.
> >>>>>>>>>
> >>>>>>>>> In other words, it's practically easier to compute spans by
> >>>>> subtracting
> >>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
> >>> spans.
> >>>>> You
> >>>>>>>>> can even just consider that the span computation from e2e
> >>> always
> >>>>> just
> >>>>>>>>> involves subtracting two numbers, whereas computing e2e latency
> >>>>> from
> >>>>>>>>> spans involves adding _all_ the spans leading up to the end you
> >>>>> care
> >>>>>>> about.
> >>>>>>>>>
> >>>>>>>>> It seems like people really prefer to have spans when they are
> >>>>>>> debugging
> >>>>>>>>> latency problems, whereas e2e latency is a more general
> >>> measurement
> >>>>>>>>> that basically every person/application cares about and should
> >>> be
> >>>>>>>>> monitoring.
> >>>>>>>>>
> >>>>>>>>> Altogether, it really seem to provide more value to more
> >>> people if
> >>>>> we
> >>>>>>>>> report
> >>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I think
> >>> I
> >>>>> have
> >>>>>>> no
> >>>>>>>>> preference, I'd defer to other peoples' intuition.
> >>>>>>>>>
> >>>>>>>>> G2:
> >>>>>>>>> I think the processor-node metric is nice, since the inside of
> >>> a
> >>>>> task
> >>>>>>> can
> >>>>>>>>> introduce a significant amount of latency in some cases. Plus,
> >>>>> it's a
> >>>>>>> more
> >>>>>>>>> direct measurement, if you really wanted to know (for the
> >>> purposes
> >>>>> of
> >>>>>>> IQ
> >>>>>>>>> or something) how long it takes source events to "show up" at
> >>> the
> >>>>>>> store.
> >>>>>>>>>
> >>>>>>>>> I think actually recording it at every processor could be
> >>>>> expensive,
> >>>>>>> but we
> >>>>>>>>> already record a bunch of metrics at the node level.
> >>>>>>>>>
> >>>>>>>>> B1:
> >>>>>>>>> I think 50% could be reasonable to record also. Even if it's a
> >>> poor
> >>>>>>> metric
> >>>>>>>>> for operational purposes, a lot of people might expect to see
> >>>>> "mean".
> >>>>>>>>> Actually,
> >>>>>>>>> I was surprised not to see "min". Is there a reason to leave it
> >>>>> off?
> >>>>>>>>>
> >>>>>>>>> I might suggest:
> >>>>>>>>> min, mean (50th), 75th, 99th, max
> >>>>>>>>>
> >>>>>>>>> B3:
> >>>>>>>>> I agree we should include late records (though not the ones we
> >>>>> drop).
> >>>>>>>>> It may be spiky, but only when there are legitimately some
> >>> records
> >>>>>>> with a
> >>>>>>>>> high end-to-end latency, which is the whole point of these
> >>> metrics.
> >>>>>>>>>
> >>>>>>>>> That's it! I don't think I have any other feedback, other than
> >>> a
> >>>>>>> request to
> >>>>>>>>> also report "min".
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> -John
> >>>>>>>>>
> >>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> >>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
> >>>>>>>>>>
> >>>>>>>>>> 1) The end-to-end latency includes both the processing
> >>> latency
> >>>>> of the
> >>>>>>>>> task
> >>>>>>>>>> and the latency spent sitting in intermediate topics. I have
> >>> a
> >>>>>>> similar
> >>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
> >>> a
> >>>>> task A
> >>>>>>>>>> actually measures the latency of the sub-topology up-to but
> >>> not
> >>>>>>> including
> >>>>>>>>>> the processing of A, which is a bit weird.
> >>>>>>>>>>
> >>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
> >>> since
> >>>>>>> today we
> >>>>>>>>>> already have several "latency" metrics already which are
> >>>>> measuring
> >>>>>>>>> elapsed
> >>>>>>>>>> system-time for processing a record / etc, while here we are
> >>>>>>> comparing
> >>>>>>>>> the
> >>>>>>>>>> system wallclock time with the record timestamp.
> >>>>>>>>>>
> >>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
> >>> (note we
> >>>>>>> already
> >>>>>>>>>> have a "record-lateness" metric), in which case recording at
> >>> the
> >>>>>>>>>> system-time before we start processing the record sounds more
> >>>>>>> natural.
> >>>>>>>>>>
> >>>>>>>>>> 2) With that in mind, I'm wondering if the
> >>> processor-node-level
> >>>>> DEBUG
> >>>>>>>>>> metric is worth to add, given that we already have a
> >>> task-level
> >>>>>>>>> processing
> >>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
> >>>>> similar
> >>>>>>> to
> >>>>>>>>> the
> >>>>>>>>>> task-level e2e latency + task-level processing latency.
> >>>>> Personally I
> >>>>>>>>> think
> >>>>>>>>>> having a task-level record-staleness metric is sufficient.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Guozhang
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> >>>>>>>>> sophie@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
> >>> this
> >>>>>>>>> specific
> >>>>>>>>>>> metric, as
> >>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
> >>> the
> >>>>>>> latency
> >>>>>>>>> you
> >>>>>>>>>>> can
> >>>>>>>>>>> reasonably expect to see. I chose percentiles that would
> >>>>> hopefully
> >>>>>>>>> give a
> >>>>>>>>>>> good
> >>>>>>>>>>> sense of what *most* records will experience, and what
> >>> *close
> >>>>> to
> >>>>>>> all*
> >>>>>>>>>>> records
> >>>>>>>>>>> will.
> >>>>>>>>>>>
> >>>>>>>>>>> However I'm not married to these specific numbers and
> >>> could be
> >>>>>>>>> convinced.
> >>>>>>>>>>> Would be especially interested in hearing from users on
> >>> this.
> >>>>>>>>>>>
> >>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
> >>>>> this KIP
> >>>>>>>>> since
> >>>>>>>>>>> users
> >>>>>>>>>>> can always compute it themselves by subtracting the
> >>> previous
> >>>>> node's
> >>>>>>>>>>> end-to-end latency. I guess we could do it either way since
> >>>>> you can
> >>>>>>>>> always
> >>>>>>>>>>> compute one from the other, but I think the end-to-end
> >>> latency
> >>>>>>> feels
> >>>>>>>>> more
> >>>>>>>>>>> valuable as it's main motivation is not to debug
> >>> bottlenecks
> >>>>> in the
> >>>>>>>>>>> topology but
> >>>>>>>>>>> to give users a sense of how long it takes arecord to be
> >>>>> reflected
> >>>>>>> in
> >>>>>>>>>>> certain parts
> >>>>>>>>>>> of the topology. For example this might be useful for users
> >>>>> who are
> >>>>>>>>>>> wondering
> >>>>>>>>>>> roughly when a record that was just produced will be
> >>> included
> >>>>> in
> >>>>>>> their
> >>>>>>>>> IQ
> >>>>>>>>>>> results.
> >>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
> >>> make
> >>>>>>> that
> >>>>>>>>> clear
> >>>>>>>>>>> enough
> >>>>>>>>>>> in the KIP's motivation.
> >>>>>>>>>>>
> >>>>>>>>>>> 3. Good question, I should address this in the KIP. The
> >>> short
> >>>>>>> answer is
> >>>>>>>>>>> "yes",
> >>>>>>>>>>> we will include late records. I added a paragraph to the
> >>> end
> >>>>> of the
> >>>>>>>>>>> Proposed
> >>>>>>>>>>> Changes section explaining the reasoning here, please let
> >>> me
> >>>>> know
> >>>>>>> if
> >>>>>>>>> you
> >>>>>>>>>>> have
> >>>>>>>>>>> any concerns.
> >>>>>>>>>>>
> >>>>>>>>>>> 4. Assuming you're referring to the existing metric
> >>>>>>> "process-latency",
> >>>>>>>>> that
> >>>>>>>>>>> metric
> >>>>>>>>>>> reflects the time for the literal Node#process method to
> >>> run
> >>>>>>> whereas
> >>>>>>>>> this
> >>>>>>>>>>> metric
> >>>>>>>>>>> would always be measured relative to the event timestamp.
> >>>>>>>>>>>
> >>>>>>>>>>> That said, the naming collision there is pretty confusing
> >>> so
> >>>>> I've
> >>>>>>>>> renamed
> >>>>>>>>>>> the
> >>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
> >>> better
> >>>>>>>>> reflects
> >>>>>>>>>>> the nature
> >>>>>>>>>>> of the metric anyway.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> >>>>>>>>> reluctanthero104@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
> >>>>> important
> >>>>>>> for
> >>>>>>>>>>>> understanding the bottleneck of the application.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A couple of questions and ideas:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
> >>> max
> >>>>>>>>> percentiles?
> >>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
> >>> well
> >>>>> in
> >>>>>>>>> production
> >>>>>>>>>>>> systems.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
> >>> if a
> >>>>>>> record
> >>>>>>>>> goes
> >>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
> >>> T(B->C) +
> >>>>>>> T(A->B) +
> >>>>>>>>>>> T(A)
> >>>>>>>>>>>> and so on, where P() represents the captured latency,
> >>> and T()
> >>>>>>>>> represents
> >>>>>>>>>>>> the time for transiting the records between two nodes,
> >>>>> including
> >>>>>>>>>>> processing
> >>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
> >>>>> T(A->B)
> >>>>>>> are
> >>>>>>>>> more
> >>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
> >>> there
> >>>>> is a
> >>>>>>>>> spike in
> >>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
> >>> time.
> >>>>> In the
> >>>>>>>>> same
> >>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
> >>> record
> >>>>> exits
> >>>>>>>>> from the
> >>>>>>>>>>>> sink as this marks the whole time this record spent
> >>> inside
> >>>>> the
> >>>>>>>>> funnel. Do
> >>>>>>>>>>>> you think we could have separate treatment for sink
> >>> nodes and
> >>>>>>> other
> >>>>>>>>>>>> nodes, so that other nodes only count the time receiving
> >>> the
> >>>>>>> record
> >>>>>>>>> from
> >>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
> >>>>> discuss
> >>>>>>>>> this
> >>>>>>>>>>>> alternative to see if it is reasonable.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. As we are going to monitor late arrival records as
> >>> well,
> >>>>> they
> >>>>>>>>> would
> >>>>>>>>>>>> create some really spiky graphs when the out-of-order
> >>>>> records are
> >>>>>>>>>>>> interleaving with on time records. Should we also supply
> >>> a
> >>>>> smooth
> >>>>>>>>> version
> >>>>>>>>>>>> of the latency metrics, or user should just take care of
> >>> it
> >>>>> by
> >>>>>>>>> themself?
> >>>>>>>>>>>>
> >>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
> >>>>> relation
> >>>>>>>>> with our
> >>>>>>>>>>>> existing processing latency metrics, could you add some
> >>>>> context
> >>>>>>> on
> >>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
> >>> the
> >>>>>>> best?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Boyang
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> >>>>>>>>> sophie@confluent.io
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
> >>> to
> >>>>> add
> >>>>>>>>> end-to-end
> >>>>>>>>>>>>> latency metrics to Streams. Please take a look:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Sophie
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> -- Guozhang
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by "Matthias J. Sax" <mj...@apache.org>.
Thanks for the KIP Sophie.

I think it's not useful to record the avg/mean; it sensitive to
outliers. We should rather track the median (50th percentile).

Not sure if tracking min is useful, but I am also ok to track it.

However, I find it odd to track 75th percentile. Standard measures would
the 90th or 95th -- I guess we don't need both, so maybe picking 90th
might be more useful?

About the name: "staleness" wound really odd, and if fact the metric
does capture "latency" so we should call it "latency". I understand the
issue that we already have a latency metric. So maybe we could call it
`record-e2e-latency-*` ?

While I agree that we should include out-or-order data (the KIP should
talk about `out-of-order` data, not `late` data; data is only `late` if
it's out-of-order and if it's dropped), I don't really understand why
the new metric would help to configure grace period or retention time?
As you mention in the KIP, both are define as max difference of
`event-time - stream-time` and thus the new metric that takes
system-/wallclock-time into account does not seem to help at all.


Btw: there is a great talk about "How NOT to Measure Latency" by Gil
Tene: https://www.youtube.com/watch?v=lJ8ydIuPFeU


-Matthias


On 5/14/20 7:17 PM, John Roesler wrote:
> Hi Sophie,
> 
> It seems like there would still be plenty of use cases for recording
> this metric at all processors and not just stateful ones, but I'm happy
> to suspend my arguments for now. Since you're proposing to keep
> them at the processor-node level, it will be seamless later to add
> in the stateless processors if we want. As a wise man once said,
> "Adding is always easier than removing."
> 
> Regarding the time measurement, it's an implementation detail
> we don't need to consider in the KIP. Nevertheless, I'd greatly
> prefer to measure the system time again when recording the
> metric. I don't think we've seen any evidence that proves this
> would harm performance, and the amount of inaccuracy using
> the cached system time could incur is actually substantial. But,
> if you want to just "not mention this" in the KIP, we can defer to
> the actual PR discussion, at which time we're in a better position
> to use benchmarks, etc., to make the call.
> 
> Along the lines of the measurement accuracy discussion, one
> minor thought I had is that maybe we should consider measuring
> the task staleness metric at the sink, rather than the source, so that
> it includes the processing latency of the task itself, not just the latency
> of everything up to, but not including, the task (which seems confusing
> for users). I guess this could also be an implementation detail, though.
> 
> Thanks for the update,
> -John
> 
> On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
>> Hey all,
>>
>> After discussing with Bruno I'd like to propose a small amendment,
>> which is to record the processor-node-level metrics only for *stateful*
>> *operators*. They would still be considered a "processor-node-level"
>> metric and not a "state-store-level" metric as the staleness is still
>> a property of the node rather than of the state itself. However, it seems
>> that this information is primarily useful for stateful operators that might
>> be exposing state via IQ or otherwise dependent on the record time
>> unlike a stateless operator.
>>
>> It's worth calling out that recent performance improvements to the metrics
>> framework mean that we no longer fetch the system time at the operator
>> level, but only once per task. In other words the system time is not updated
>> between each process as a record flows through the subtopology, so
>> debugging the processor-level latency via the stateleness will not be
>> possible.Note that this doesn't mean the operator-level metrics are not
>> *useful* relative to the task-level metric. Upstream caching and/or
>> suppression
>> can still cause a record's staleness at some downstream stateful operator
>> to deviate from the task-level staleness (recorded at the source node).
>>
>> Please let me know if you have any concerns about this change. The
>> KIP has been updated with the new proposal
>>
>> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io> wrote:
>>
>>> Hi Sophie,
>>>
>>> Thank you for the KIP.
>>>
>>> The KIP looks good to me.
>>>
>>> 50th percentile:
>>> I think we do not need it now. If we need it, we can add it. Here the
>>> old truism applies: Adding is always easier than removing.
>>>
>>> processor-node-level metrics:
>>> I think it is good to have the staleness metrics also on
>>> processor-node-level. If we do not want to record them on all
>>> processor nodes, you could restrict the recording to stateful
>>> processor-nodes, since those are the ones that would benefit most from
>>> the staleness metrics.
>>>
>>> Best,
>>> Bruno
>>>
>>> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <so...@confluent.io>
>>> wrote:
>>>>
>>>> Yeah, the specific reason was just to align with the current metrics.
>>>>
>>>> Is it better to conform than to be right? History has a lot to say on
>>> that
>>>> matter
>>>> but I'm not sure how much of it applies to the fine details of metrics
>>>> naming :P
>>>>
>>>> More seriously, I figured if people are looking at this metric they're
>>>> likely to
>>>> be looking at all the others. Then naming this one "-mean" would probably
>>>> lead some to conclude that the "-avg" suffix in the other metrics has a
>>>> different meaning.
>>>>
>>>> As for the percentiles, I actually like p99 (and p75) better. I'll swap
>>>> that out
>>>>
>>>> On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org>
>>> wrote:
>>>>
>>>>> Thanks Sophie,
>>>>>
>>>>> I hope this isn't too nit-picky, but is there a reason to choose "avg"
>>>>> instead
>>>>> of "mean"? Maybe this is too paranoid, and I might be oversensitive
>>> because
>>>>> of the mistake I just made earlier, but it strikes me that "avg" is
>>>>> actually
>>>>> ambiguous, as it refers to a family of statistics, whereas "mean" is
>>>>> specific.
>>>>> I see other Kafka metrics with "avg", but none with "mean"; was that
>>> the
>>>>> reason? If so, I'm +1.
>>>>>
>>>>> Regarding the names of the percentile, I actually couldn't find _any_
>>> other
>>>>> metrics that use percentile. Was there a reason to choose "99th" as
>>> opposed
>>>>> to "p99" or any other scheme? This is not a criticism, I'm just
>>> primarily
>>>>> asking
>>>>> for consistency's sake.
>>>>>
>>>>> Thanks again,
>>>>> -John
>>>>>
>>>>> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
>>>>>> Alright, I can get behind adding the min metric for the sake of
>>> pretty
>>>>>> graphs
>>>>>> (and trivial computation).
>>>>>>
>>>>>> I'm still on the fence regarding the mean (or 50th percentile) but I
>>> can
>>>>> see
>>>>>> how users might expect it and find it a bit disorienting not to
>>> have. So
>>>>> the
>>>>>> updated proposed metrics are
>>>>>>
>>>>>>
>>>>>>    - record-staleness-max [ms]
>>>>>>    - record-staleness-99th [ms] *(99th percentile)*
>>>>>>    - record-staleness-75th [ms] *(75th percentile)*
>>>>>>    - record-staleness-avg [ms] *(mean)*
>>>>>>    - record-staleness-min [ms]
>>>>>>
>>>>>>
>>>>>> On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>>> Oh boy, I never miss an opportunity to embarrass myself. I guess
>>> the
>>>>> mean
>>>>>>> seems more interesting to me than the median, but neither are as
>>>>>>> interesting as the higher percentiles (99th and max).
>>>>>>>
>>>>>>> Min isn’t really important for any SLAs, but it does round out the
>>>>> mental
>>>>>>> picture of the distribution. I’ve always graphed min along with the
>>>>> other
>>>>>>> metrics to help me understand how fast the system can be, which
>>> helps
>>>>> in
>>>>>>> optimization decisions. It’s also a relatively inexpensive metric
>>> to
>>>>>>> compute, so it might be nice to just throw it in.
>>>>>>>
>>>>>>> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
>>>>>>>> G1:
>>>>>>>> I was considering it as the "end-to-end latency *up* to the
>>> specific
>>>>>>> task"
>>>>>>>> but
>>>>>>>> I'm happy with "record-staleness" if that drives the point home
>>>>> better.
>>>>>>> So
>>>>>>>> it's the
>>>>>>>> "staleness of the record when it is received by that task" --
>>> will
>>>>> update
>>>>>>>> the KIP
>>>>>>>>
>>>>>>>> B1/J:
>>>>>>>> I'm struggling to imagine a case where the min would actually be
>>>>> useful,
>>>>>>>> rather than
>>>>>>>> just intellectually interesting. I don't feel strongly that we
>>>>> shouldn't
>>>>>>>> add it, but that's
>>>>>>>> why I didn't include it from the start. Can you enlighten me
>>> with an
>>>>>>>> example?
>>>>>>>>
>>>>>>>> I was also vaguely concerned about the overhead of adding
>>> multiple
>>>>>>>> percentile
>>>>>>>> metrics. Do we have any data to indicate what kind of performance
>>>>> hit we
>>>>>>>> take on
>>>>>>>> metrics computation?
>>>>>>>>
>>>>>>>> Also, not to be too pedantic but the 50th percentile would be the
>>>>> median
>>>>>>>> not the
>>>>>>>> mean. Would you propose to add the mean *and* the 50th
>>> percentile, or
>>>>>>> just
>>>>>>>> one
>>>>>>>> of the two?
>>>>>>>>
>>>>>>>> Thanks all!
>>>>>>>> Sophie
>>>>>>>>
>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John Roesler <
>>> vvcephei@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello all, and thanks for the KIP, Sophie,
>>>>>>>>>
>>>>>>>>> Just some comments on the discussion so far:
>>>>>>>>>
>>>>>>>>> B2/G1:
>>>>>>>>> In principle, it shouldn't matter whether we report "spans" or
>>>>>>>>> "end-to-end" latency. But in practice, some of the spans are
>>> pretty
>>>>>>>>> difficult to really measure (like time spent waiting in the
>>>>> topics, or
>>>>>>>>> time from the event happening to the ETL producer choosing to
>>> send
>>>>> it,
>>>>>>>>> or time spent in send/receive buffers, etc., etc.
>>>>>>>>>
>>>>>>>>> In other words, it's practically easier to compute spans by
>>>>> subtracting
>>>>>>>>> e2e latencies than it is to compute e2e latencies by adding
>>> spans.
>>>>> You
>>>>>>>>> can even just consider that the span computation from e2e
>>> always
>>>>> just
>>>>>>>>> involves subtracting two numbers, whereas computing e2e latency
>>>>> from
>>>>>>>>> spans involves adding _all_ the spans leading up to the end you
>>>>> care
>>>>>>> about.
>>>>>>>>>
>>>>>>>>> It seems like people really prefer to have spans when they are
>>>>>>> debugging
>>>>>>>>> latency problems, whereas e2e latency is a more general
>>> measurement
>>>>>>>>> that basically every person/application cares about and should
>>> be
>>>>>>>>> monitoring.
>>>>>>>>>
>>>>>>>>> Altogether, it really seem to provide more value to more
>>> people if
>>>>> we
>>>>>>>>> report
>>>>>>>>> e2e latencies. Regarding "record-staleness" as a name, I think
>>> I
>>>>> have
>>>>>>> no
>>>>>>>>> preference, I'd defer to other peoples' intuition.
>>>>>>>>>
>>>>>>>>> G2:
>>>>>>>>> I think the processor-node metric is nice, since the inside of
>>> a
>>>>> task
>>>>>>> can
>>>>>>>>> introduce a significant amount of latency in some cases. Plus,
>>>>> it's a
>>>>>>> more
>>>>>>>>> direct measurement, if you really wanted to know (for the
>>> purposes
>>>>> of
>>>>>>> IQ
>>>>>>>>> or something) how long it takes source events to "show up" at
>>> the
>>>>>>> store.
>>>>>>>>>
>>>>>>>>> I think actually recording it at every processor could be
>>>>> expensive,
>>>>>>> but we
>>>>>>>>> already record a bunch of metrics at the node level.
>>>>>>>>>
>>>>>>>>> B1:
>>>>>>>>> I think 50% could be reasonable to record also. Even if it's a
>>> poor
>>>>>>> metric
>>>>>>>>> for operational purposes, a lot of people might expect to see
>>>>> "mean".
>>>>>>>>> Actually,
>>>>>>>>> I was surprised not to see "min". Is there a reason to leave it
>>>>> off?
>>>>>>>>>
>>>>>>>>> I might suggest:
>>>>>>>>> min, mean (50th), 75th, 99th, max
>>>>>>>>>
>>>>>>>>> B3:
>>>>>>>>> I agree we should include late records (though not the ones we
>>>>> drop).
>>>>>>>>> It may be spiky, but only when there are legitimately some
>>> records
>>>>>>> with a
>>>>>>>>> high end-to-end latency, which is the whole point of these
>>> metrics.
>>>>>>>>>
>>>>>>>>> That's it! I don't think I have any other feedback, other than
>>> a
>>>>>>> request to
>>>>>>>>> also report "min".
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
>>>>>>>>>> Thanks Sophie for the KIP, a few quick thoughts:
>>>>>>>>>>
>>>>>>>>>> 1) The end-to-end latency includes both the processing
>>> latency
>>>>> of the
>>>>>>>>> task
>>>>>>>>>> and the latency spent sitting in intermediate topics. I have
>>> a
>>>>>>> similar
>>>>>>>>>> feeling as Boyang mentioned above that the latency metric of
>>> a
>>>>> task A
>>>>>>>>>> actually measures the latency of the sub-topology up-to but
>>> not
>>>>>>> including
>>>>>>>>>> the processing of A, which is a bit weird.
>>>>>>>>>>
>>>>>>>>>> Maybe the my feeling comes from the name "latency" itself,
>>> since
>>>>>>> today we
>>>>>>>>>> already have several "latency" metrics already which are
>>>>> measuring
>>>>>>>>> elapsed
>>>>>>>>>> system-time for processing a record / etc, while here we are
>>>>>>> comparing
>>>>>>>>> the
>>>>>>>>>> system wallclock time with the record timestamp.
>>>>>>>>>>
>>>>>>>>>> Maybe we can consider renaming it as "record-staleness"
>>> (note we
>>>>>>> already
>>>>>>>>>> have a "record-lateness" metric), in which case recording at
>>> the
>>>>>>>>>> system-time before we start processing the record sounds more
>>>>>>> natural.
>>>>>>>>>>
>>>>>>>>>> 2) With that in mind, I'm wondering if the
>>> processor-node-level
>>>>> DEBUG
>>>>>>>>>> metric is worth to add, given that we already have a
>>> task-level
>>>>>>>>> processing
>>>>>>>>>> latency metric. Basically, a specific node's e2e latency is
>>>>> similar
>>>>>>> to
>>>>>>>>> the
>>>>>>>>>> task-level e2e latency + task-level processing latency.
>>>>> Personally I
>>>>>>>>> think
>>>>>>>>>> having a task-level record-staleness metric is sufficient.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Guozhang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
>>>>>>>>> sophie@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> 1. I felt that 50% was not a particularly useful gauge for
>>> this
>>>>>>>>> specific
>>>>>>>>>>> metric, as
>>>>>>>>>>> it's presumably most useful at putting an *upper *bound on
>>> the
>>>>>>> latency
>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>> reasonably expect to see. I chose percentiles that would
>>>>> hopefully
>>>>>>>>> give a
>>>>>>>>>>> good
>>>>>>>>>>> sense of what *most* records will experience, and what
>>> *close
>>>>> to
>>>>>>> all*
>>>>>>>>>>> records
>>>>>>>>>>> will.
>>>>>>>>>>>
>>>>>>>>>>> However I'm not married to these specific numbers and
>>> could be
>>>>>>>>> convinced.
>>>>>>>>>>> Would be especially interested in hearing from users on
>>> this.
>>>>>>>>>>>
>>>>>>>>>>> 2. I'm inclined to not include the "hop-to-hop latency" in
>>>>> this KIP
>>>>>>>>> since
>>>>>>>>>>> users
>>>>>>>>>>> can always compute it themselves by subtracting the
>>> previous
>>>>> node's
>>>>>>>>>>> end-to-end latency. I guess we could do it either way since
>>>>> you can
>>>>>>>>> always
>>>>>>>>>>> compute one from the other, but I think the end-to-end
>>> latency
>>>>>>> feels
>>>>>>>>> more
>>>>>>>>>>> valuable as it's main motivation is not to debug
>>> bottlenecks
>>>>> in the
>>>>>>>>>>> topology but
>>>>>>>>>>> to give users a sense of how long it takes arecord to be
>>>>> reflected
>>>>>>> in
>>>>>>>>>>> certain parts
>>>>>>>>>>> of the topology. For example this might be useful for users
>>>>> who are
>>>>>>>>>>> wondering
>>>>>>>>>>> roughly when a record that was just produced will be
>>> included
>>>>> in
>>>>>>> their
>>>>>>>>> IQ
>>>>>>>>>>> results.
>>>>>>>>>>> Debugging is just a nice side effect -- but maybe I didn't
>>> make
>>>>>>> that
>>>>>>>>> clear
>>>>>>>>>>> enough
>>>>>>>>>>> in the KIP's motivation.
>>>>>>>>>>>
>>>>>>>>>>> 3. Good question, I should address this in the KIP. The
>>> short
>>>>>>> answer is
>>>>>>>>>>> "yes",
>>>>>>>>>>> we will include late records. I added a paragraph to the
>>> end
>>>>> of the
>>>>>>>>>>> Proposed
>>>>>>>>>>> Changes section explaining the reasoning here, please let
>>> me
>>>>> know
>>>>>>> if
>>>>>>>>> you
>>>>>>>>>>> have
>>>>>>>>>>> any concerns.
>>>>>>>>>>>
>>>>>>>>>>> 4. Assuming you're referring to the existing metric
>>>>>>> "process-latency",
>>>>>>>>> that
>>>>>>>>>>> metric
>>>>>>>>>>> reflects the time for the literal Node#process method to
>>> run
>>>>>>> whereas
>>>>>>>>> this
>>>>>>>>>>> metric
>>>>>>>>>>> would always be measured relative to the event timestamp.
>>>>>>>>>>>
>>>>>>>>>>> That said, the naming collision there is pretty confusing
>>> so
>>>>> I've
>>>>>>>>> renamed
>>>>>>>>>>> the
>>>>>>>>>>> metrics in this KIP to "end-to-end-latency" which I feel
>>> better
>>>>>>>>> reflects
>>>>>>>>>>> the nature
>>>>>>>>>>> of the metric anyway.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
>>>>>>>>> reluctanthero104@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the KIP Sophie. Getting the E2E latency is
>>>>> important
>>>>>>> for
>>>>>>>>>>>> understanding the bottleneck of the application.
>>>>>>>>>>>>
>>>>>>>>>>>> A couple of questions and ideas:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Could you clarify the rational of picking 75, 99 and
>>> max
>>>>>>>>> percentiles?
>>>>>>>>>>>> Normally I see cases where we use 50, 90 percentile as
>>> well
>>>>> in
>>>>>>>>> production
>>>>>>>>>>>> systems.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. The current latency being computed is cumulative, I.E
>>> if a
>>>>>>> record
>>>>>>>>> goes
>>>>>>>>>>>> through A -> B -> C, then P(C) = T(B->C) + P(B) =
>>> T(B->C) +
>>>>>>> T(A->B) +
>>>>>>>>>>> T(A)
>>>>>>>>>>>> and so on, where P() represents the captured latency,
>>> and T()
>>>>>>>>> represents
>>>>>>>>>>>> the time for transiting the records between two nodes,
>>>>> including
>>>>>>>>>>> processing
>>>>>>>>>>>> time. For monitoring purpose, maybe having T(B->C) and
>>>>> T(A->B)
>>>>>>> are
>>>>>>>>> more
>>>>>>>>>>>> natural to view as "hop-to-hop latency", otherwise if
>>> there
>>>>> is a
>>>>>>>>> spike in
>>>>>>>>>>>> T(A->B), both P(B) and P(C) are affected in the same
>>> time.
>>>>> In the
>>>>>>>>> same
>>>>>>>>>>>> spirit, the E2E latency is meaningful only when the
>>> record
>>>>> exits
>>>>>>>>> from the
>>>>>>>>>>>> sink as this marks the whole time this record spent
>>> inside
>>>>> the
>>>>>>>>> funnel. Do
>>>>>>>>>>>> you think we could have separate treatment for sink
>>> nodes and
>>>>>>> other
>>>>>>>>>>>> nodes, so that other nodes only count the time receiving
>>> the
>>>>>>> record
>>>>>>>>> from
>>>>>>>>>>>> last hop? I'm not proposing a solution here, just want to
>>>>> discuss
>>>>>>>>> this
>>>>>>>>>>>> alternative to see if it is reasonable.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. As we are going to monitor late arrival records as
>>> well,
>>>>> they
>>>>>>>>> would
>>>>>>>>>>>> create some really spiky graphs when the out-of-order
>>>>> records are
>>>>>>>>>>>> interleaving with on time records. Should we also supply
>>> a
>>>>> smooth
>>>>>>>>> version
>>>>>>>>>>>> of the latency metrics, or user should just take care of
>>> it
>>>>> by
>>>>>>>>> themself?
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Regarding this new metrics, we haven't discussed its
>>>>> relation
>>>>>>>>> with our
>>>>>>>>>>>> existing processing latency metrics, could you add some
>>>>> context
>>>>>>> on
>>>>>>>>>>>> comparison and a simple `when to use which` tutorial for
>>> the
>>>>>>> best?
>>>>>>>>>>>>
>>>>>>>>>>>> Boyang
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
>>>>>>>>> sophie@confluent.io
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to kick off discussion on KIP-613 which aims
>>> to
>>>>> add
>>>>>>>>> end-to-end
>>>>>>>>>>>>> latency metrics to Streams. Please take a look:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Sophie
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -- Guozhang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>


Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
Hi Sophie,

It seems like there would still be plenty of use cases for recording
this metric at all processors and not just stateful ones, but I'm happy
to suspend my arguments for now. Since you're proposing to keep
them at the processor-node level, it will be seamless later to add
in the stateless processors if we want. As a wise man once said,
"Adding is always easier than removing."

Regarding the time measurement, it's an implementation detail
we don't need to consider in the KIP. Nevertheless, I'd greatly
prefer to measure the system time again when recording the
metric. I don't think we've seen any evidence that proves this
would harm performance, and the amount of inaccuracy using
the cached system time could incur is actually substantial. But,
if you want to just "not mention this" in the KIP, we can defer to
the actual PR discussion, at which time we're in a better position
to use benchmarks, etc., to make the call.

Along the lines of the measurement accuracy discussion, one
minor thought I had is that maybe we should consider measuring
the task staleness metric at the sink, rather than the source, so that
it includes the processing latency of the task itself, not just the latency
of everything up to, but not including, the task (which seems confusing
for users). I guess this could also be an implementation detail, though.

Thanks for the update,
-John

On Thu, May 14, 2020, at 13:31, Sophie Blee-Goldman wrote:
> Hey all,
> 
> After discussing with Bruno I'd like to propose a small amendment,
> which is to record the processor-node-level metrics only for *stateful*
> *operators*. They would still be considered a "processor-node-level"
> metric and not a "state-store-level" metric as the staleness is still
> a property of the node rather than of the state itself. However, it seems
> that this information is primarily useful for stateful operators that might
> be exposing state via IQ or otherwise dependent on the record time
> unlike a stateless operator.
> 
> It's worth calling out that recent performance improvements to the metrics
> framework mean that we no longer fetch the system time at the operator
> level, but only once per task. In other words the system time is not updated
> between each process as a record flows through the subtopology, so
> debugging the processor-level latency via the stateleness will not be
> possible.Note that this doesn't mean the operator-level metrics are not
> *useful* relative to the task-level metric. Upstream caching and/or
> suppression
> can still cause a record's staleness at some downstream stateful operator
> to deviate from the task-level staleness (recorded at the source node).
> 
> Please let me know if you have any concerns about this change. The
> KIP has been updated with the new proposal
> 
> On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io> wrote:
> 
> > Hi Sophie,
> >
> > Thank you for the KIP.
> >
> > The KIP looks good to me.
> >
> > 50th percentile:
> > I think we do not need it now. If we need it, we can add it. Here the
> > old truism applies: Adding is always easier than removing.
> >
> > processor-node-level metrics:
> > I think it is good to have the staleness metrics also on
> > processor-node-level. If we do not want to record them on all
> > processor nodes, you could restrict the recording to stateful
> > processor-nodes, since those are the ones that would benefit most from
> > the staleness metrics.
> >
> > Best,
> > Bruno
> >
> > On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <so...@confluent.io>
> > wrote:
> > >
> > > Yeah, the specific reason was just to align with the current metrics.
> > >
> > > Is it better to conform than to be right? History has a lot to say on
> > that
> > > matter
> > > but I'm not sure how much of it applies to the fine details of metrics
> > > naming :P
> > >
> > > More seriously, I figured if people are looking at this metric they're
> > > likely to
> > > be looking at all the others. Then naming this one "-mean" would probably
> > > lead some to conclude that the "-avg" suffix in the other metrics has a
> > > different meaning.
> > >
> > > As for the percentiles, I actually like p99 (and p75) better. I'll swap
> > > that out
> > >
> > > On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > Thanks Sophie,
> > > >
> > > > I hope this isn't too nit-picky, but is there a reason to choose "avg"
> > > > instead
> > > > of "mean"? Maybe this is too paranoid, and I might be oversensitive
> > because
> > > > of the mistake I just made earlier, but it strikes me that "avg" is
> > > > actually
> > > > ambiguous, as it refers to a family of statistics, whereas "mean" is
> > > > specific.
> > > > I see other Kafka metrics with "avg", but none with "mean"; was that
> > the
> > > > reason? If so, I'm +1.
> > > >
> > > > Regarding the names of the percentile, I actually couldn't find _any_
> > other
> > > > metrics that use percentile. Was there a reason to choose "99th" as
> > opposed
> > > > to "p99" or any other scheme? This is not a criticism, I'm just
> > primarily
> > > > asking
> > > > for consistency's sake.
> > > >
> > > > Thanks again,
> > > > -John
> > > >
> > > > On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > > > Alright, I can get behind adding the min metric for the sake of
> > pretty
> > > > > graphs
> > > > > (and trivial computation).
> > > > >
> > > > > I'm still on the fence regarding the mean (or 50th percentile) but I
> > can
> > > > see
> > > > > how users might expect it and find it a bit disorienting not to
> > have. So
> > > > the
> > > > > updated proposed metrics are
> > > > >
> > > > >
> > > > >    - record-staleness-max [ms]
> > > > >    - record-staleness-99th [ms] *(99th percentile)*
> > > > >    - record-staleness-75th [ms] *(75th percentile)*
> > > > >    - record-staleness-avg [ms] *(mean)*
> > > > >    - record-staleness-min [ms]
> > > > >
> > > > >
> > > > > On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Oh boy, I never miss an opportunity to embarrass myself. I guess
> > the
> > > > mean
> > > > > > seems more interesting to me than the median, but neither are as
> > > > > > interesting as the higher percentiles (99th and max).
> > > > > >
> > > > > > Min isn’t really important for any SLAs, but it does round out the
> > > > mental
> > > > > > picture of the distribution. I’ve always graphed min along with the
> > > > other
> > > > > > metrics to help me understand how fast the system can be, which
> > helps
> > > > in
> > > > > > optimization decisions. It’s also a relatively inexpensive metric
> > to
> > > > > > compute, so it might be nice to just throw it in.
> > > > > >
> > > > > > On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > > > > > G1:
> > > > > > > I was considering it as the "end-to-end latency *up* to the
> > specific
> > > > > > task"
> > > > > > > but
> > > > > > > I'm happy with "record-staleness" if that drives the point home
> > > > better.
> > > > > > So
> > > > > > > it's the
> > > > > > > "staleness of the record when it is received by that task" --
> > will
> > > > update
> > > > > > > the KIP
> > > > > > >
> > > > > > > B1/J:
> > > > > > > I'm struggling to imagine a case where the min would actually be
> > > > useful,
> > > > > > > rather than
> > > > > > > just intellectually interesting. I don't feel strongly that we
> > > > shouldn't
> > > > > > > add it, but that's
> > > > > > > why I didn't include it from the start. Can you enlighten me
> > with an
> > > > > > > example?
> > > > > > >
> > > > > > > I was also vaguely concerned about the overhead of adding
> > multiple
> > > > > > > percentile
> > > > > > > metrics. Do we have any data to indicate what kind of performance
> > > > hit we
> > > > > > > take on
> > > > > > > metrics computation?
> > > > > > >
> > > > > > > Also, not to be too pedantic but the 50th percentile would be the
> > > > median
> > > > > > > not the
> > > > > > > mean. Would you propose to add the mean *and* the 50th
> > percentile, or
> > > > > > just
> > > > > > > one
> > > > > > > of the two?
> > > > > > >
> > > > > > > Thanks all!
> > > > > > > Sophie
> > > > > > >
> > > > > > > On Wed, May 13, 2020 at 3:34 PM John Roesler <
> > vvcephei@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello all, and thanks for the KIP, Sophie,
> > > > > > > >
> > > > > > > > Just some comments on the discussion so far:
> > > > > > > >
> > > > > > > > B2/G1:
> > > > > > > > In principle, it shouldn't matter whether we report "spans" or
> > > > > > > > "end-to-end" latency. But in practice, some of the spans are
> > pretty
> > > > > > > > difficult to really measure (like time spent waiting in the
> > > > topics, or
> > > > > > > > time from the event happening to the ETL producer choosing to
> > send
> > > > it,
> > > > > > > > or time spent in send/receive buffers, etc., etc.
> > > > > > > >
> > > > > > > > In other words, it's practically easier to compute spans by
> > > > subtracting
> > > > > > > > e2e latencies than it is to compute e2e latencies by adding
> > spans.
> > > > You
> > > > > > > > can even just consider that the span computation from e2e
> > always
> > > > just
> > > > > > > > involves subtracting two numbers, whereas computing e2e latency
> > > > from
> > > > > > > > spans involves adding _all_ the spans leading up to the end you
> > > > care
> > > > > > about.
> > > > > > > >
> > > > > > > > It seems like people really prefer to have spans when they are
> > > > > > debugging
> > > > > > > > latency problems, whereas e2e latency is a more general
> > measurement
> > > > > > > > that basically every person/application cares about and should
> > be
> > > > > > > > monitoring.
> > > > > > > >
> > > > > > > > Altogether, it really seem to provide more value to more
> > people if
> > > > we
> > > > > > > > report
> > > > > > > > e2e latencies. Regarding "record-staleness" as a name, I think
> > I
> > > > have
> > > > > > no
> > > > > > > > preference, I'd defer to other peoples' intuition.
> > > > > > > >
> > > > > > > > G2:
> > > > > > > > I think the processor-node metric is nice, since the inside of
> > a
> > > > task
> > > > > > can
> > > > > > > > introduce a significant amount of latency in some cases. Plus,
> > > > it's a
> > > > > > more
> > > > > > > > direct measurement, if you really wanted to know (for the
> > purposes
> > > > of
> > > > > > IQ
> > > > > > > > or something) how long it takes source events to "show up" at
> > the
> > > > > > store.
> > > > > > > >
> > > > > > > > I think actually recording it at every processor could be
> > > > expensive,
> > > > > > but we
> > > > > > > > already record a bunch of metrics at the node level.
> > > > > > > >
> > > > > > > > B1:
> > > > > > > > I think 50% could be reasonable to record also. Even if it's a
> > poor
> > > > > > metric
> > > > > > > > for operational purposes, a lot of people might expect to see
> > > > "mean".
> > > > > > > > Actually,
> > > > > > > > I was surprised not to see "min". Is there a reason to leave it
> > > > off?
> > > > > > > >
> > > > > > > > I might suggest:
> > > > > > > > min, mean (50th), 75th, 99th, max
> > > > > > > >
> > > > > > > > B3:
> > > > > > > > I agree we should include late records (though not the ones we
> > > > drop).
> > > > > > > > It may be spiky, but only when there are legitimately some
> > records
> > > > > > with a
> > > > > > > > high end-to-end latency, which is the whole point of these
> > metrics.
> > > > > > > >
> > > > > > > > That's it! I don't think I have any other feedback, other than
> > a
> > > > > > request to
> > > > > > > > also report "min".
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -John
> > > > > > > >
> > > > > > > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > > > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > > > > > > >
> > > > > > > > > 1) The end-to-end latency includes both the processing
> > latency
> > > > of the
> > > > > > > > task
> > > > > > > > > and the latency spent sitting in intermediate topics. I have
> > a
> > > > > > similar
> > > > > > > > > feeling as Boyang mentioned above that the latency metric of
> > a
> > > > task A
> > > > > > > > > actually measures the latency of the sub-topology up-to but
> > not
> > > > > > including
> > > > > > > > > the processing of A, which is a bit weird.
> > > > > > > > >
> > > > > > > > > Maybe the my feeling comes from the name "latency" itself,
> > since
> > > > > > today we
> > > > > > > > > already have several "latency" metrics already which are
> > > > measuring
> > > > > > > > elapsed
> > > > > > > > > system-time for processing a record / etc, while here we are
> > > > > > comparing
> > > > > > > > the
> > > > > > > > > system wallclock time with the record timestamp.
> > > > > > > > >
> > > > > > > > > Maybe we can consider renaming it as "record-staleness"
> > (note we
> > > > > > already
> > > > > > > > > have a "record-lateness" metric), in which case recording at
> > the
> > > > > > > > > system-time before we start processing the record sounds more
> > > > > > natural.
> > > > > > > > >
> > > > > > > > > 2) With that in mind, I'm wondering if the
> > processor-node-level
> > > > DEBUG
> > > > > > > > > metric is worth to add, given that we already have a
> > task-level
> > > > > > > > processing
> > > > > > > > > latency metric. Basically, a specific node's e2e latency is
> > > > similar
> > > > > > to
> > > > > > > > the
> > > > > > > > > task-level e2e latency + task-level processing latency.
> > > > Personally I
> > > > > > > > think
> > > > > > > > > having a task-level record-staleness metric is sufficient.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > > > > > sophie@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > 1. I felt that 50% was not a particularly useful gauge for
> > this
> > > > > > > > specific
> > > > > > > > > > metric, as
> > > > > > > > > > it's presumably most useful at putting an *upper *bound on
> > the
> > > > > > latency
> > > > > > > > you
> > > > > > > > > > can
> > > > > > > > > > reasonably expect to see. I chose percentiles that would
> > > > hopefully
> > > > > > > > give a
> > > > > > > > > > good
> > > > > > > > > > sense of what *most* records will experience, and what
> > *close
> > > > to
> > > > > > all*
> > > > > > > > > > records
> > > > > > > > > > will.
> > > > > > > > > >
> > > > > > > > > > However I'm not married to these specific numbers and
> > could be
> > > > > > > > convinced.
> > > > > > > > > > Would be especially interested in hearing from users on
> > this.
> > > > > > > > > >
> > > > > > > > > > 2. I'm inclined to not include the "hop-to-hop latency" in
> > > > this KIP
> > > > > > > > since
> > > > > > > > > > users
> > > > > > > > > > can always compute it themselves by subtracting the
> > previous
> > > > node's
> > > > > > > > > > end-to-end latency. I guess we could do it either way since
> > > > you can
> > > > > > > > always
> > > > > > > > > > compute one from the other, but I think the end-to-end
> > latency
> > > > > > feels
> > > > > > > > more
> > > > > > > > > > valuable as it's main motivation is not to debug
> > bottlenecks
> > > > in the
> > > > > > > > > > topology but
> > > > > > > > > > to give users a sense of how long it takes arecord to be
> > > > reflected
> > > > > > in
> > > > > > > > > > certain parts
> > > > > > > > > > of the topology. For example this might be useful for users
> > > > who are
> > > > > > > > > > wondering
> > > > > > > > > > roughly when a record that was just produced will be
> > included
> > > > in
> > > > > > their
> > > > > > > > IQ
> > > > > > > > > > results.
> > > > > > > > > > Debugging is just a nice side effect -- but maybe I didn't
> > make
> > > > > > that
> > > > > > > > clear
> > > > > > > > > > enough
> > > > > > > > > > in the KIP's motivation.
> > > > > > > > > >
> > > > > > > > > > 3. Good question, I should address this in the KIP. The
> > short
> > > > > > answer is
> > > > > > > > > > "yes",
> > > > > > > > > > we will include late records. I added a paragraph to the
> > end
> > > > of the
> > > > > > > > > > Proposed
> > > > > > > > > > Changes section explaining the reasoning here, please let
> > me
> > > > know
> > > > > > if
> > > > > > > > you
> > > > > > > > > > have
> > > > > > > > > > any concerns.
> > > > > > > > > >
> > > > > > > > > > 4. Assuming you're referring to the existing metric
> > > > > > "process-latency",
> > > > > > > > that
> > > > > > > > > > metric
> > > > > > > > > > reflects the time for the literal Node#process method to
> > run
> > > > > > whereas
> > > > > > > > this
> > > > > > > > > > metric
> > > > > > > > > > would always be measured relative to the event timestamp.
> > > > > > > > > >
> > > > > > > > > > That said, the naming collision there is pretty confusing
> > so
> > > > I've
> > > > > > > > renamed
> > > > > > > > > > the
> > > > > > > > > > metrics in this KIP to "end-to-end-latency" which I feel
> > better
> > > > > > > > reflects
> > > > > > > > > > the nature
> > > > > > > > > > of the metric anyway.
> > > > > > > > > >
> > > > > > > > > > Thanks for the feedback!
> > > > > > > > > >
> > > > > > > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > > > > reluctanthero104@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the KIP Sophie. Getting the E2E latency is
> > > > important
> > > > > > for
> > > > > > > > > > > understanding the bottleneck of the application.
> > > > > > > > > > >
> > > > > > > > > > > A couple of questions and ideas:
> > > > > > > > > > >
> > > > > > > > > > > 1. Could you clarify the rational of picking 75, 99 and
> > max
> > > > > > > > percentiles?
> > > > > > > > > > > Normally I see cases where we use 50, 90 percentile as
> > well
> > > > in
> > > > > > > > production
> > > > > > > > > > > systems.
> > > > > > > > > > >
> > > > > > > > > > > 2. The current latency being computed is cumulative, I.E
> > if a
> > > > > > record
> > > > > > > > goes
> > > > > > > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) =
> > T(B->C) +
> > > > > > T(A->B) +
> > > > > > > > > > T(A)
> > > > > > > > > > > and so on, where P() represents the captured latency,
> > and T()
> > > > > > > > represents
> > > > > > > > > > > the time for transiting the records between two nodes,
> > > > including
> > > > > > > > > > processing
> > > > > > > > > > > time. For monitoring purpose, maybe having T(B->C) and
> > > > T(A->B)
> > > > > > are
> > > > > > > > more
> > > > > > > > > > > natural to view as "hop-to-hop latency", otherwise if
> > there
> > > > is a
> > > > > > > > spike in
> > > > > > > > > > > T(A->B), both P(B) and P(C) are affected in the same
> > time.
> > > > In the
> > > > > > > > same
> > > > > > > > > > > spirit, the E2E latency is meaningful only when the
> > record
> > > > exits
> > > > > > > > from the
> > > > > > > > > > > sink as this marks the whole time this record spent
> > inside
> > > > the
> > > > > > > > funnel. Do
> > > > > > > > > > > you think we could have separate treatment for sink
> > nodes and
> > > > > > other
> > > > > > > > > > > nodes, so that other nodes only count the time receiving
> > the
> > > > > > record
> > > > > > > > from
> > > > > > > > > > > last hop? I'm not proposing a solution here, just want to
> > > > discuss
> > > > > > > > this
> > > > > > > > > > > alternative to see if it is reasonable.
> > > > > > > > > > >
> > > > > > > > > > > 3. As we are going to monitor late arrival records as
> > well,
> > > > they
> > > > > > > > would
> > > > > > > > > > > create some really spiky graphs when the out-of-order
> > > > records are
> > > > > > > > > > > interleaving with on time records. Should we also supply
> > a
> > > > smooth
> > > > > > > > version
> > > > > > > > > > > of the latency metrics, or user should just take care of
> > it
> > > > by
> > > > > > > > themself?
> > > > > > > > > > >
> > > > > > > > > > > 4. Regarding this new metrics, we haven't discussed its
> > > > relation
> > > > > > > > with our
> > > > > > > > > > > existing processing latency metrics, could you add some
> > > > context
> > > > > > on
> > > > > > > > > > > comparison and a simple `when to use which` tutorial for
> > the
> > > > > > best?
> > > > > > > > > > >
> > > > > > > > > > > Boyang
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > > > > > > sophie@confluent.io
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to kick off discussion on KIP-613 which aims
> > to
> > > > add
> > > > > > > > end-to-end
> > > > > > > > > > > > latency metrics to Streams. Please take a look:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Sophie
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Hey all,

After discussing with Bruno I'd like to propose a small amendment,
which is to record the processor-node-level metrics only for *stateful*
*operators*. They would still be considered a "processor-node-level"
metric and not a "state-store-level" metric as the staleness is still
a property of the node rather than of the state itself. However, it seems
that this information is primarily useful for stateful operators that might
be exposing state via IQ or otherwise dependent on the record time
unlike a stateless operator.

It's worth calling out that recent performance improvements to the metrics
framework mean that we no longer fetch the system time at the operator
level, but only once per task. In other words the system time is not updated
between each process as a record flows through the subtopology, so
debugging the processor-level latency via the stateleness will not be
possible.Note that this doesn't mean the operator-level metrics are not
*useful* relative to the task-level metric. Upstream caching and/or
suppression
can still cause a record's staleness at some downstream stateful operator
to deviate from the task-level staleness (recorded at the source node).

Please let me know if you have any concerns about this change. The
KIP has been updated with the new proposal

On Thu, May 14, 2020 at 3:04 AM Bruno Cadonna <br...@confluent.io> wrote:

> Hi Sophie,
>
> Thank you for the KIP.
>
> The KIP looks good to me.
>
> 50th percentile:
> I think we do not need it now. If we need it, we can add it. Here the
> old truism applies: Adding is always easier than removing.
>
> processor-node-level metrics:
> I think it is good to have the staleness metrics also on
> processor-node-level. If we do not want to record them on all
> processor nodes, you could restrict the recording to stateful
> processor-nodes, since those are the ones that would benefit most from
> the staleness metrics.
>
> Best,
> Bruno
>
> On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
> >
> > Yeah, the specific reason was just to align with the current metrics.
> >
> > Is it better to conform than to be right? History has a lot to say on
> that
> > matter
> > but I'm not sure how much of it applies to the fine details of metrics
> > naming :P
> >
> > More seriously, I figured if people are looking at this metric they're
> > likely to
> > be looking at all the others. Then naming this one "-mean" would probably
> > lead some to conclude that the "-avg" suffix in the other metrics has a
> > different meaning.
> >
> > As for the percentiles, I actually like p99 (and p75) better. I'll swap
> > that out
> >
> > On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org>
> wrote:
> >
> > > Thanks Sophie,
> > >
> > > I hope this isn't too nit-picky, but is there a reason to choose "avg"
> > > instead
> > > of "mean"? Maybe this is too paranoid, and I might be oversensitive
> because
> > > of the mistake I just made earlier, but it strikes me that "avg" is
> > > actually
> > > ambiguous, as it refers to a family of statistics, whereas "mean" is
> > > specific.
> > > I see other Kafka metrics with "avg", but none with "mean"; was that
> the
> > > reason? If so, I'm +1.
> > >
> > > Regarding the names of the percentile, I actually couldn't find _any_
> other
> > > metrics that use percentile. Was there a reason to choose "99th" as
> opposed
> > > to "p99" or any other scheme? This is not a criticism, I'm just
> primarily
> > > asking
> > > for consistency's sake.
> > >
> > > Thanks again,
> > > -John
> > >
> > > On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > > Alright, I can get behind adding the min metric for the sake of
> pretty
> > > > graphs
> > > > (and trivial computation).
> > > >
> > > > I'm still on the fence regarding the mean (or 50th percentile) but I
> can
> > > see
> > > > how users might expect it and find it a bit disorienting not to
> have. So
> > > the
> > > > updated proposed metrics are
> > > >
> > > >
> > > >    - record-staleness-max [ms]
> > > >    - record-staleness-99th [ms] *(99th percentile)*
> > > >    - record-staleness-75th [ms] *(75th percentile)*
> > > >    - record-staleness-avg [ms] *(mean)*
> > > >    - record-staleness-min [ms]
> > > >
> > > >
> > > > On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > > > Oh boy, I never miss an opportunity to embarrass myself. I guess
> the
> > > mean
> > > > > seems more interesting to me than the median, but neither are as
> > > > > interesting as the higher percentiles (99th and max).
> > > > >
> > > > > Min isn’t really important for any SLAs, but it does round out the
> > > mental
> > > > > picture of the distribution. I’ve always graphed min along with the
> > > other
> > > > > metrics to help me understand how fast the system can be, which
> helps
> > > in
> > > > > optimization decisions. It’s also a relatively inexpensive metric
> to
> > > > > compute, so it might be nice to just throw it in.
> > > > >
> > > > > On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > > > > G1:
> > > > > > I was considering it as the "end-to-end latency *up* to the
> specific
> > > > > task"
> > > > > > but
> > > > > > I'm happy with "record-staleness" if that drives the point home
> > > better.
> > > > > So
> > > > > > it's the
> > > > > > "staleness of the record when it is received by that task" --
> will
> > > update
> > > > > > the KIP
> > > > > >
> > > > > > B1/J:
> > > > > > I'm struggling to imagine a case where the min would actually be
> > > useful,
> > > > > > rather than
> > > > > > just intellectually interesting. I don't feel strongly that we
> > > shouldn't
> > > > > > add it, but that's
> > > > > > why I didn't include it from the start. Can you enlighten me
> with an
> > > > > > example?
> > > > > >
> > > > > > I was also vaguely concerned about the overhead of adding
> multiple
> > > > > > percentile
> > > > > > metrics. Do we have any data to indicate what kind of performance
> > > hit we
> > > > > > take on
> > > > > > metrics computation?
> > > > > >
> > > > > > Also, not to be too pedantic but the 50th percentile would be the
> > > median
> > > > > > not the
> > > > > > mean. Would you propose to add the mean *and* the 50th
> percentile, or
> > > > > just
> > > > > > one
> > > > > > of the two?
> > > > > >
> > > > > > Thanks all!
> > > > > > Sophie
> > > > > >
> > > > > > On Wed, May 13, 2020 at 3:34 PM John Roesler <
> vvcephei@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hello all, and thanks for the KIP, Sophie,
> > > > > > >
> > > > > > > Just some comments on the discussion so far:
> > > > > > >
> > > > > > > B2/G1:
> > > > > > > In principle, it shouldn't matter whether we report "spans" or
> > > > > > > "end-to-end" latency. But in practice, some of the spans are
> pretty
> > > > > > > difficult to really measure (like time spent waiting in the
> > > topics, or
> > > > > > > time from the event happening to the ETL producer choosing to
> send
> > > it,
> > > > > > > or time spent in send/receive buffers, etc., etc.
> > > > > > >
> > > > > > > In other words, it's practically easier to compute spans by
> > > subtracting
> > > > > > > e2e latencies than it is to compute e2e latencies by adding
> spans.
> > > You
> > > > > > > can even just consider that the span computation from e2e
> always
> > > just
> > > > > > > involves subtracting two numbers, whereas computing e2e latency
> > > from
> > > > > > > spans involves adding _all_ the spans leading up to the end you
> > > care
> > > > > about.
> > > > > > >
> > > > > > > It seems like people really prefer to have spans when they are
> > > > > debugging
> > > > > > > latency problems, whereas e2e latency is a more general
> measurement
> > > > > > > that basically every person/application cares about and should
> be
> > > > > > > monitoring.
> > > > > > >
> > > > > > > Altogether, it really seem to provide more value to more
> people if
> > > we
> > > > > > > report
> > > > > > > e2e latencies. Regarding "record-staleness" as a name, I think
> I
> > > have
> > > > > no
> > > > > > > preference, I'd defer to other peoples' intuition.
> > > > > > >
> > > > > > > G2:
> > > > > > > I think the processor-node metric is nice, since the inside of
> a
> > > task
> > > > > can
> > > > > > > introduce a significant amount of latency in some cases. Plus,
> > > it's a
> > > > > more
> > > > > > > direct measurement, if you really wanted to know (for the
> purposes
> > > of
> > > > > IQ
> > > > > > > or something) how long it takes source events to "show up" at
> the
> > > > > store.
> > > > > > >
> > > > > > > I think actually recording it at every processor could be
> > > expensive,
> > > > > but we
> > > > > > > already record a bunch of metrics at the node level.
> > > > > > >
> > > > > > > B1:
> > > > > > > I think 50% could be reasonable to record also. Even if it's a
> poor
> > > > > metric
> > > > > > > for operational purposes, a lot of people might expect to see
> > > "mean".
> > > > > > > Actually,
> > > > > > > I was surprised not to see "min". Is there a reason to leave it
> > > off?
> > > > > > >
> > > > > > > I might suggest:
> > > > > > > min, mean (50th), 75th, 99th, max
> > > > > > >
> > > > > > > B3:
> > > > > > > I agree we should include late records (though not the ones we
> > > drop).
> > > > > > > It may be spiky, but only when there are legitimately some
> records
> > > > > with a
> > > > > > > high end-to-end latency, which is the whole point of these
> metrics.
> > > > > > >
> > > > > > > That's it! I don't think I have any other feedback, other than
> a
> > > > > request to
> > > > > > > also report "min".
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -John
> > > > > > >
> > > > > > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > > > > > >
> > > > > > > > 1) The end-to-end latency includes both the processing
> latency
> > > of the
> > > > > > > task
> > > > > > > > and the latency spent sitting in intermediate topics. I have
> a
> > > > > similar
> > > > > > > > feeling as Boyang mentioned above that the latency metric of
> a
> > > task A
> > > > > > > > actually measures the latency of the sub-topology up-to but
> not
> > > > > including
> > > > > > > > the processing of A, which is a bit weird.
> > > > > > > >
> > > > > > > > Maybe the my feeling comes from the name "latency" itself,
> since
> > > > > today we
> > > > > > > > already have several "latency" metrics already which are
> > > measuring
> > > > > > > elapsed
> > > > > > > > system-time for processing a record / etc, while here we are
> > > > > comparing
> > > > > > > the
> > > > > > > > system wallclock time with the record timestamp.
> > > > > > > >
> > > > > > > > Maybe we can consider renaming it as "record-staleness"
> (note we
> > > > > already
> > > > > > > > have a "record-lateness" metric), in which case recording at
> the
> > > > > > > > system-time before we start processing the record sounds more
> > > > > natural.
> > > > > > > >
> > > > > > > > 2) With that in mind, I'm wondering if the
> processor-node-level
> > > DEBUG
> > > > > > > > metric is worth to add, given that we already have a
> task-level
> > > > > > > processing
> > > > > > > > latency metric. Basically, a specific node's e2e latency is
> > > similar
> > > > > to
> > > > > > > the
> > > > > > > > task-level e2e latency + task-level processing latency.
> > > Personally I
> > > > > > > think
> > > > > > > > having a task-level record-staleness metric is sufficient.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > > > > sophie@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > 1. I felt that 50% was not a particularly useful gauge for
> this
> > > > > > > specific
> > > > > > > > > metric, as
> > > > > > > > > it's presumably most useful at putting an *upper *bound on
> the
> > > > > latency
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > reasonably expect to see. I chose percentiles that would
> > > hopefully
> > > > > > > give a
> > > > > > > > > good
> > > > > > > > > sense of what *most* records will experience, and what
> *close
> > > to
> > > > > all*
> > > > > > > > > records
> > > > > > > > > will.
> > > > > > > > >
> > > > > > > > > However I'm not married to these specific numbers and
> could be
> > > > > > > convinced.
> > > > > > > > > Would be especially interested in hearing from users on
> this.
> > > > > > > > >
> > > > > > > > > 2. I'm inclined to not include the "hop-to-hop latency" in
> > > this KIP
> > > > > > > since
> > > > > > > > > users
> > > > > > > > > can always compute it themselves by subtracting the
> previous
> > > node's
> > > > > > > > > end-to-end latency. I guess we could do it either way since
> > > you can
> > > > > > > always
> > > > > > > > > compute one from the other, but I think the end-to-end
> latency
> > > > > feels
> > > > > > > more
> > > > > > > > > valuable as it's main motivation is not to debug
> bottlenecks
> > > in the
> > > > > > > > > topology but
> > > > > > > > > to give users a sense of how long it takes arecord to be
> > > reflected
> > > > > in
> > > > > > > > > certain parts
> > > > > > > > > of the topology. For example this might be useful for users
> > > who are
> > > > > > > > > wondering
> > > > > > > > > roughly when a record that was just produced will be
> included
> > > in
> > > > > their
> > > > > > > IQ
> > > > > > > > > results.
> > > > > > > > > Debugging is just a nice side effect -- but maybe I didn't
> make
> > > > > that
> > > > > > > clear
> > > > > > > > > enough
> > > > > > > > > in the KIP's motivation.
> > > > > > > > >
> > > > > > > > > 3. Good question, I should address this in the KIP. The
> short
> > > > > answer is
> > > > > > > > > "yes",
> > > > > > > > > we will include late records. I added a paragraph to the
> end
> > > of the
> > > > > > > > > Proposed
> > > > > > > > > Changes section explaining the reasoning here, please let
> me
> > > know
> > > > > if
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > any concerns.
> > > > > > > > >
> > > > > > > > > 4. Assuming you're referring to the existing metric
> > > > > "process-latency",
> > > > > > > that
> > > > > > > > > metric
> > > > > > > > > reflects the time for the literal Node#process method to
> run
> > > > > whereas
> > > > > > > this
> > > > > > > > > metric
> > > > > > > > > would always be measured relative to the event timestamp.
> > > > > > > > >
> > > > > > > > > That said, the naming collision there is pretty confusing
> so
> > > I've
> > > > > > > renamed
> > > > > > > > > the
> > > > > > > > > metrics in this KIP to "end-to-end-latency" which I feel
> better
> > > > > > > reflects
> > > > > > > > > the nature
> > > > > > > > > of the metric anyway.
> > > > > > > > >
> > > > > > > > > Thanks for the feedback!
> > > > > > > > >
> > > > > > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > > > reluctanthero104@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the KIP Sophie. Getting the E2E latency is
> > > important
> > > > > for
> > > > > > > > > > understanding the bottleneck of the application.
> > > > > > > > > >
> > > > > > > > > > A couple of questions and ideas:
> > > > > > > > > >
> > > > > > > > > > 1. Could you clarify the rational of picking 75, 99 and
> max
> > > > > > > percentiles?
> > > > > > > > > > Normally I see cases where we use 50, 90 percentile as
> well
> > > in
> > > > > > > production
> > > > > > > > > > systems.
> > > > > > > > > >
> > > > > > > > > > 2. The current latency being computed is cumulative, I.E
> if a
> > > > > record
> > > > > > > goes
> > > > > > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) =
> T(B->C) +
> > > > > T(A->B) +
> > > > > > > > > T(A)
> > > > > > > > > > and so on, where P() represents the captured latency,
> and T()
> > > > > > > represents
> > > > > > > > > > the time for transiting the records between two nodes,
> > > including
> > > > > > > > > processing
> > > > > > > > > > time. For monitoring purpose, maybe having T(B->C) and
> > > T(A->B)
> > > > > are
> > > > > > > more
> > > > > > > > > > natural to view as "hop-to-hop latency", otherwise if
> there
> > > is a
> > > > > > > spike in
> > > > > > > > > > T(A->B), both P(B) and P(C) are affected in the same
> time.
> > > In the
> > > > > > > same
> > > > > > > > > > spirit, the E2E latency is meaningful only when the
> record
> > > exits
> > > > > > > from the
> > > > > > > > > > sink as this marks the whole time this record spent
> inside
> > > the
> > > > > > > funnel. Do
> > > > > > > > > > you think we could have separate treatment for sink
> nodes and
> > > > > other
> > > > > > > > > > nodes, so that other nodes only count the time receiving
> the
> > > > > record
> > > > > > > from
> > > > > > > > > > last hop? I'm not proposing a solution here, just want to
> > > discuss
> > > > > > > this
> > > > > > > > > > alternative to see if it is reasonable.
> > > > > > > > > >
> > > > > > > > > > 3. As we are going to monitor late arrival records as
> well,
> > > they
> > > > > > > would
> > > > > > > > > > create some really spiky graphs when the out-of-order
> > > records are
> > > > > > > > > > interleaving with on time records. Should we also supply
> a
> > > smooth
> > > > > > > version
> > > > > > > > > > of the latency metrics, or user should just take care of
> it
> > > by
> > > > > > > themself?
> > > > > > > > > >
> > > > > > > > > > 4. Regarding this new metrics, we haven't discussed its
> > > relation
> > > > > > > with our
> > > > > > > > > > existing processing latency metrics, could you add some
> > > context
> > > > > on
> > > > > > > > > > comparison and a simple `when to use which` tutorial for
> the
> > > > > best?
> > > > > > > > > >
> > > > > > > > > > Boyang
> > > > > > > > > >
> > > > > > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > > > > > sophie@confluent.io
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to kick off discussion on KIP-613 which aims
> to
> > > add
> > > > > > > end-to-end
> > > > > > > > > > > latency metrics to Streams. Please take a look:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Sophie
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Bruno Cadonna <br...@confluent.io>.
Hi Sophie,

Thank you for the KIP.

The KIP looks good to me.

50th percentile:
I think we do not need it now. If we need it, we can add it. Here the
old truism applies: Adding is always easier than removing.

processor-node-level metrics:
I think it is good to have the staleness metrics also on
processor-node-level. If we do not want to record them on all
processor nodes, you could restrict the recording to stateful
processor-nodes, since those are the ones that would benefit most from
the staleness metrics.

Best,
Bruno

On Thu, May 14, 2020 at 4:15 AM Sophie Blee-Goldman <so...@confluent.io> wrote:
>
> Yeah, the specific reason was just to align with the current metrics.
>
> Is it better to conform than to be right? History has a lot to say on that
> matter
> but I'm not sure how much of it applies to the fine details of metrics
> naming :P
>
> More seriously, I figured if people are looking at this metric they're
> likely to
> be looking at all the others. Then naming this one "-mean" would probably
> lead some to conclude that the "-avg" suffix in the other metrics has a
> different meaning.
>
> As for the percentiles, I actually like p99 (and p75) better. I'll swap
> that out
>
> On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org> wrote:
>
> > Thanks Sophie,
> >
> > I hope this isn't too nit-picky, but is there a reason to choose "avg"
> > instead
> > of "mean"? Maybe this is too paranoid, and I might be oversensitive because
> > of the mistake I just made earlier, but it strikes me that "avg" is
> > actually
> > ambiguous, as it refers to a family of statistics, whereas "mean" is
> > specific.
> > I see other Kafka metrics with "avg", but none with "mean"; was that the
> > reason? If so, I'm +1.
> >
> > Regarding the names of the percentile, I actually couldn't find _any_ other
> > metrics that use percentile. Was there a reason to choose "99th" as opposed
> > to "p99" or any other scheme? This is not a criticism, I'm just primarily
> > asking
> > for consistency's sake.
> >
> > Thanks again,
> > -John
> >
> > On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > > Alright, I can get behind adding the min metric for the sake of pretty
> > > graphs
> > > (and trivial computation).
> > >
> > > I'm still on the fence regarding the mean (or 50th percentile) but I can
> > see
> > > how users might expect it and find it a bit disorienting not to have. So
> > the
> > > updated proposed metrics are
> > >
> > >
> > >    - record-staleness-max [ms]
> > >    - record-staleness-99th [ms] *(99th percentile)*
> > >    - record-staleness-75th [ms] *(75th percentile)*
> > >    - record-staleness-avg [ms] *(mean)*
> > >    - record-staleness-min [ms]
> > >
> > >
> > > On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > Oh boy, I never miss an opportunity to embarrass myself. I guess the
> > mean
> > > > seems more interesting to me than the median, but neither are as
> > > > interesting as the higher percentiles (99th and max).
> > > >
> > > > Min isn’t really important for any SLAs, but it does round out the
> > mental
> > > > picture of the distribution. I’ve always graphed min along with the
> > other
> > > > metrics to help me understand how fast the system can be, which helps
> > in
> > > > optimization decisions. It’s also a relatively inexpensive metric to
> > > > compute, so it might be nice to just throw it in.
> > > >
> > > > On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > > > G1:
> > > > > I was considering it as the "end-to-end latency *up* to the specific
> > > > task"
> > > > > but
> > > > > I'm happy with "record-staleness" if that drives the point home
> > better.
> > > > So
> > > > > it's the
> > > > > "staleness of the record when it is received by that task" -- will
> > update
> > > > > the KIP
> > > > >
> > > > > B1/J:
> > > > > I'm struggling to imagine a case where the min would actually be
> > useful,
> > > > > rather than
> > > > > just intellectually interesting. I don't feel strongly that we
> > shouldn't
> > > > > add it, but that's
> > > > > why I didn't include it from the start. Can you enlighten me with an
> > > > > example?
> > > > >
> > > > > I was also vaguely concerned about the overhead of adding multiple
> > > > > percentile
> > > > > metrics. Do we have any data to indicate what kind of performance
> > hit we
> > > > > take on
> > > > > metrics computation?
> > > > >
> > > > > Also, not to be too pedantic but the 50th percentile would be the
> > median
> > > > > not the
> > > > > mean. Would you propose to add the mean *and* the 50th percentile, or
> > > > just
> > > > > one
> > > > > of the two?
> > > > >
> > > > > Thanks all!
> > > > > Sophie
> > > > >
> > > > > On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hello all, and thanks for the KIP, Sophie,
> > > > > >
> > > > > > Just some comments on the discussion so far:
> > > > > >
> > > > > > B2/G1:
> > > > > > In principle, it shouldn't matter whether we report "spans" or
> > > > > > "end-to-end" latency. But in practice, some of the spans are pretty
> > > > > > difficult to really measure (like time spent waiting in the
> > topics, or
> > > > > > time from the event happening to the ETL producer choosing to send
> > it,
> > > > > > or time spent in send/receive buffers, etc., etc.
> > > > > >
> > > > > > In other words, it's practically easier to compute spans by
> > subtracting
> > > > > > e2e latencies than it is to compute e2e latencies by adding spans.
> > You
> > > > > > can even just consider that the span computation from e2e always
> > just
> > > > > > involves subtracting two numbers, whereas computing e2e latency
> > from
> > > > > > spans involves adding _all_ the spans leading up to the end you
> > care
> > > > about.
> > > > > >
> > > > > > It seems like people really prefer to have spans when they are
> > > > debugging
> > > > > > latency problems, whereas e2e latency is a more general measurement
> > > > > > that basically every person/application cares about and should be
> > > > > > monitoring.
> > > > > >
> > > > > > Altogether, it really seem to provide more value to more people if
> > we
> > > > > > report
> > > > > > e2e latencies. Regarding "record-staleness" as a name, I think I
> > have
> > > > no
> > > > > > preference, I'd defer to other peoples' intuition.
> > > > > >
> > > > > > G2:
> > > > > > I think the processor-node metric is nice, since the inside of a
> > task
> > > > can
> > > > > > introduce a significant amount of latency in some cases. Plus,
> > it's a
> > > > more
> > > > > > direct measurement, if you really wanted to know (for the purposes
> > of
> > > > IQ
> > > > > > or something) how long it takes source events to "show up" at the
> > > > store.
> > > > > >
> > > > > > I think actually recording it at every processor could be
> > expensive,
> > > > but we
> > > > > > already record a bunch of metrics at the node level.
> > > > > >
> > > > > > B1:
> > > > > > I think 50% could be reasonable to record also. Even if it's a poor
> > > > metric
> > > > > > for operational purposes, a lot of people might expect to see
> > "mean".
> > > > > > Actually,
> > > > > > I was surprised not to see "min". Is there a reason to leave it
> > off?
> > > > > >
> > > > > > I might suggest:
> > > > > > min, mean (50th), 75th, 99th, max
> > > > > >
> > > > > > B3:
> > > > > > I agree we should include late records (though not the ones we
> > drop).
> > > > > > It may be spiky, but only when there are legitimately some records
> > > > with a
> > > > > > high end-to-end latency, which is the whole point of these metrics.
> > > > > >
> > > > > > That's it! I don't think I have any other feedback, other than a
> > > > request to
> > > > > > also report "min".
> > > > > >
> > > > > > Thanks,
> > > > > > -John
> > > > > >
> > > > > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > > > > >
> > > > > > > 1) The end-to-end latency includes both the processing latency
> > of the
> > > > > > task
> > > > > > > and the latency spent sitting in intermediate topics. I have a
> > > > similar
> > > > > > > feeling as Boyang mentioned above that the latency metric of a
> > task A
> > > > > > > actually measures the latency of the sub-topology up-to but not
> > > > including
> > > > > > > the processing of A, which is a bit weird.
> > > > > > >
> > > > > > > Maybe the my feeling comes from the name "latency" itself, since
> > > > today we
> > > > > > > already have several "latency" metrics already which are
> > measuring
> > > > > > elapsed
> > > > > > > system-time for processing a record / etc, while here we are
> > > > comparing
> > > > > > the
> > > > > > > system wallclock time with the record timestamp.
> > > > > > >
> > > > > > > Maybe we can consider renaming it as "record-staleness" (note we
> > > > already
> > > > > > > have a "record-lateness" metric), in which case recording at the
> > > > > > > system-time before we start processing the record sounds more
> > > > natural.
> > > > > > >
> > > > > > > 2) With that in mind, I'm wondering if the processor-node-level
> > DEBUG
> > > > > > > metric is worth to add, given that we already have a task-level
> > > > > > processing
> > > > > > > latency metric. Basically, a specific node's e2e latency is
> > similar
> > > > to
> > > > > > the
> > > > > > > task-level e2e latency + task-level processing latency.
> > Personally I
> > > > > > think
> > > > > > > having a task-level record-staleness metric is sufficient.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > > > sophie@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > 1. I felt that 50% was not a particularly useful gauge for this
> > > > > > specific
> > > > > > > > metric, as
> > > > > > > > it's presumably most useful at putting an *upper *bound on the
> > > > latency
> > > > > > you
> > > > > > > > can
> > > > > > > > reasonably expect to see. I chose percentiles that would
> > hopefully
> > > > > > give a
> > > > > > > > good
> > > > > > > > sense of what *most* records will experience, and what *close
> > to
> > > > all*
> > > > > > > > records
> > > > > > > > will.
> > > > > > > >
> > > > > > > > However I'm not married to these specific numbers and could be
> > > > > > convinced.
> > > > > > > > Would be especially interested in hearing from users on this.
> > > > > > > >
> > > > > > > > 2. I'm inclined to not include the "hop-to-hop latency" in
> > this KIP
> > > > > > since
> > > > > > > > users
> > > > > > > > can always compute it themselves by subtracting the previous
> > node's
> > > > > > > > end-to-end latency. I guess we could do it either way since
> > you can
> > > > > > always
> > > > > > > > compute one from the other, but I think the end-to-end latency
> > > > feels
> > > > > > more
> > > > > > > > valuable as it's main motivation is not to debug bottlenecks
> > in the
> > > > > > > > topology but
> > > > > > > > to give users a sense of how long it takes arecord to be
> > reflected
> > > > in
> > > > > > > > certain parts
> > > > > > > > of the topology. For example this might be useful for users
> > who are
> > > > > > > > wondering
> > > > > > > > roughly when a record that was just produced will be included
> > in
> > > > their
> > > > > > IQ
> > > > > > > > results.
> > > > > > > > Debugging is just a nice side effect -- but maybe I didn't make
> > > > that
> > > > > > clear
> > > > > > > > enough
> > > > > > > > in the KIP's motivation.
> > > > > > > >
> > > > > > > > 3. Good question, I should address this in the KIP. The short
> > > > answer is
> > > > > > > > "yes",
> > > > > > > > we will include late records. I added a paragraph to the end
> > of the
> > > > > > > > Proposed
> > > > > > > > Changes section explaining the reasoning here, please let me
> > know
> > > > if
> > > > > > you
> > > > > > > > have
> > > > > > > > any concerns.
> > > > > > > >
> > > > > > > > 4. Assuming you're referring to the existing metric
> > > > "process-latency",
> > > > > > that
> > > > > > > > metric
> > > > > > > > reflects the time for the literal Node#process method to run
> > > > whereas
> > > > > > this
> > > > > > > > metric
> > > > > > > > would always be measured relative to the event timestamp.
> > > > > > > >
> > > > > > > > That said, the naming collision there is pretty confusing so
> > I've
> > > > > > renamed
> > > > > > > > the
> > > > > > > > metrics in this KIP to "end-to-end-latency" which I feel better
> > > > > > reflects
> > > > > > > > the nature
> > > > > > > > of the metric anyway.
> > > > > > > >
> > > > > > > > Thanks for the feedback!
> > > > > > > >
> > > > > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > > reluctanthero104@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the KIP Sophie. Getting the E2E latency is
> > important
> > > > for
> > > > > > > > > understanding the bottleneck of the application.
> > > > > > > > >
> > > > > > > > > A couple of questions and ideas:
> > > > > > > > >
> > > > > > > > > 1. Could you clarify the rational of picking 75, 99 and max
> > > > > > percentiles?
> > > > > > > > > Normally I see cases where we use 50, 90 percentile as well
> > in
> > > > > > production
> > > > > > > > > systems.
> > > > > > > > >
> > > > > > > > > 2. The current latency being computed is cumulative, I.E if a
> > > > record
> > > > > > goes
> > > > > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) +
> > > > T(A->B) +
> > > > > > > > T(A)
> > > > > > > > > and so on, where P() represents the captured latency, and T()
> > > > > > represents
> > > > > > > > > the time for transiting the records between two nodes,
> > including
> > > > > > > > processing
> > > > > > > > > time. For monitoring purpose, maybe having T(B->C) and
> > T(A->B)
> > > > are
> > > > > > more
> > > > > > > > > natural to view as "hop-to-hop latency", otherwise if there
> > is a
> > > > > > spike in
> > > > > > > > > T(A->B), both P(B) and P(C) are affected in the same time.
> > In the
> > > > > > same
> > > > > > > > > spirit, the E2E latency is meaningful only when the record
> > exits
> > > > > > from the
> > > > > > > > > sink as this marks the whole time this record spent inside
> > the
> > > > > > funnel. Do
> > > > > > > > > you think we could have separate treatment for sink nodes and
> > > > other
> > > > > > > > > nodes, so that other nodes only count the time receiving the
> > > > record
> > > > > > from
> > > > > > > > > last hop? I'm not proposing a solution here, just want to
> > discuss
> > > > > > this
> > > > > > > > > alternative to see if it is reasonable.
> > > > > > > > >
> > > > > > > > > 3. As we are going to monitor late arrival records as well,
> > they
> > > > > > would
> > > > > > > > > create some really spiky graphs when the out-of-order
> > records are
> > > > > > > > > interleaving with on time records. Should we also supply a
> > smooth
> > > > > > version
> > > > > > > > > of the latency metrics, or user should just take care of it
> > by
> > > > > > themself?
> > > > > > > > >
> > > > > > > > > 4. Regarding this new metrics, we haven't discussed its
> > relation
> > > > > > with our
> > > > > > > > > existing processing latency metrics, could you add some
> > context
> > > > on
> > > > > > > > > comparison and a simple `when to use which` tutorial for the
> > > > best?
> > > > > > > > >
> > > > > > > > > Boyang
> > > > > > > > >
> > > > > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > > > > sophie@confluent.io
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey all,
> > > > > > > > > >
> > > > > > > > > > I'd like to kick off discussion on KIP-613 which aims to
> > add
> > > > > > end-to-end
> > > > > > > > > > latency metrics to Streams. Please take a look:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Sophie
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Yeah, the specific reason was just to align with the current metrics.

Is it better to conform than to be right? History has a lot to say on that
matter
but I'm not sure how much of it applies to the fine details of metrics
naming :P

More seriously, I figured if people are looking at this metric they're
likely to
be looking at all the others. Then naming this one "-mean" would probably
lead some to conclude that the "-avg" suffix in the other metrics has a
different meaning.

As for the percentiles, I actually like p99 (and p75) better. I'll swap
that out

On Wed, May 13, 2020 at 7:07 PM John Roesler <vv...@apache.org> wrote:

> Thanks Sophie,
>
> I hope this isn't too nit-picky, but is there a reason to choose "avg"
> instead
> of "mean"? Maybe this is too paranoid, and I might be oversensitive because
> of the mistake I just made earlier, but it strikes me that "avg" is
> actually
> ambiguous, as it refers to a family of statistics, whereas "mean" is
> specific.
> I see other Kafka metrics with "avg", but none with "mean"; was that the
> reason? If so, I'm +1.
>
> Regarding the names of the percentile, I actually couldn't find _any_ other
> metrics that use percentile. Was there a reason to choose "99th" as opposed
> to "p99" or any other scheme? This is not a criticism, I'm just primarily
> asking
> for consistency's sake.
>
> Thanks again,
> -John
>
> On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> > Alright, I can get behind adding the min metric for the sake of pretty
> > graphs
> > (and trivial computation).
> >
> > I'm still on the fence regarding the mean (or 50th percentile) but I can
> see
> > how users might expect it and find it a bit disorienting not to have. So
> the
> > updated proposed metrics are
> >
> >
> >    - record-staleness-max [ms]
> >    - record-staleness-99th [ms] *(99th percentile)*
> >    - record-staleness-75th [ms] *(75th percentile)*
> >    - record-staleness-avg [ms] *(mean)*
> >    - record-staleness-min [ms]
> >
> >
> > On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org>
> wrote:
> >
> > > Oh boy, I never miss an opportunity to embarrass myself. I guess the
> mean
> > > seems more interesting to me than the median, but neither are as
> > > interesting as the higher percentiles (99th and max).
> > >
> > > Min isn’t really important for any SLAs, but it does round out the
> mental
> > > picture of the distribution. I’ve always graphed min along with the
> other
> > > metrics to help me understand how fast the system can be, which helps
> in
> > > optimization decisions. It’s also a relatively inexpensive metric to
> > > compute, so it might be nice to just throw it in.
> > >
> > > On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > > G1:
> > > > I was considering it as the "end-to-end latency *up* to the specific
> > > task"
> > > > but
> > > > I'm happy with "record-staleness" if that drives the point home
> better.
> > > So
> > > > it's the
> > > > "staleness of the record when it is received by that task" -- will
> update
> > > > the KIP
> > > >
> > > > B1/J:
> > > > I'm struggling to imagine a case where the min would actually be
> useful,
> > > > rather than
> > > > just intellectually interesting. I don't feel strongly that we
> shouldn't
> > > > add it, but that's
> > > > why I didn't include it from the start. Can you enlighten me with an
> > > > example?
> > > >
> > > > I was also vaguely concerned about the overhead of adding multiple
> > > > percentile
> > > > metrics. Do we have any data to indicate what kind of performance
> hit we
> > > > take on
> > > > metrics computation?
> > > >
> > > > Also, not to be too pedantic but the 50th percentile would be the
> median
> > > > not the
> > > > mean. Would you propose to add the mean *and* the 50th percentile, or
> > > just
> > > > one
> > > > of the two?
> > > >
> > > > Thanks all!
> > > > Sophie
> > > >
> > > > On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > > > Hello all, and thanks for the KIP, Sophie,
> > > > >
> > > > > Just some comments on the discussion so far:
> > > > >
> > > > > B2/G1:
> > > > > In principle, it shouldn't matter whether we report "spans" or
> > > > > "end-to-end" latency. But in practice, some of the spans are pretty
> > > > > difficult to really measure (like time spent waiting in the
> topics, or
> > > > > time from the event happening to the ETL producer choosing to send
> it,
> > > > > or time spent in send/receive buffers, etc., etc.
> > > > >
> > > > > In other words, it's practically easier to compute spans by
> subtracting
> > > > > e2e latencies than it is to compute e2e latencies by adding spans.
> You
> > > > > can even just consider that the span computation from e2e always
> just
> > > > > involves subtracting two numbers, whereas computing e2e latency
> from
> > > > > spans involves adding _all_ the spans leading up to the end you
> care
> > > about.
> > > > >
> > > > > It seems like people really prefer to have spans when they are
> > > debugging
> > > > > latency problems, whereas e2e latency is a more general measurement
> > > > > that basically every person/application cares about and should be
> > > > > monitoring.
> > > > >
> > > > > Altogether, it really seem to provide more value to more people if
> we
> > > > > report
> > > > > e2e latencies. Regarding "record-staleness" as a name, I think I
> have
> > > no
> > > > > preference, I'd defer to other peoples' intuition.
> > > > >
> > > > > G2:
> > > > > I think the processor-node metric is nice, since the inside of a
> task
> > > can
> > > > > introduce a significant amount of latency in some cases. Plus,
> it's a
> > > more
> > > > > direct measurement, if you really wanted to know (for the purposes
> of
> > > IQ
> > > > > or something) how long it takes source events to "show up" at the
> > > store.
> > > > >
> > > > > I think actually recording it at every processor could be
> expensive,
> > > but we
> > > > > already record a bunch of metrics at the node level.
> > > > >
> > > > > B1:
> > > > > I think 50% could be reasonable to record also. Even if it's a poor
> > > metric
> > > > > for operational purposes, a lot of people might expect to see
> "mean".
> > > > > Actually,
> > > > > I was surprised not to see "min". Is there a reason to leave it
> off?
> > > > >
> > > > > I might suggest:
> > > > > min, mean (50th), 75th, 99th, max
> > > > >
> > > > > B3:
> > > > > I agree we should include late records (though not the ones we
> drop).
> > > > > It may be spiky, but only when there are legitimately some records
> > > with a
> > > > > high end-to-end latency, which is the whole point of these metrics.
> > > > >
> > > > > That's it! I don't think I have any other feedback, other than a
> > > request to
> > > > > also report "min".
> > > > >
> > > > > Thanks,
> > > > > -John
> > > > >
> > > > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > > > >
> > > > > > 1) The end-to-end latency includes both the processing latency
> of the
> > > > > task
> > > > > > and the latency spent sitting in intermediate topics. I have a
> > > similar
> > > > > > feeling as Boyang mentioned above that the latency metric of a
> task A
> > > > > > actually measures the latency of the sub-topology up-to but not
> > > including
> > > > > > the processing of A, which is a bit weird.
> > > > > >
> > > > > > Maybe the my feeling comes from the name "latency" itself, since
> > > today we
> > > > > > already have several "latency" metrics already which are
> measuring
> > > > > elapsed
> > > > > > system-time for processing a record / etc, while here we are
> > > comparing
> > > > > the
> > > > > > system wallclock time with the record timestamp.
> > > > > >
> > > > > > Maybe we can consider renaming it as "record-staleness" (note we
> > > already
> > > > > > have a "record-lateness" metric), in which case recording at the
> > > > > > system-time before we start processing the record sounds more
> > > natural.
> > > > > >
> > > > > > 2) With that in mind, I'm wondering if the processor-node-level
> DEBUG
> > > > > > metric is worth to add, given that we already have a task-level
> > > > > processing
> > > > > > latency metric. Basically, a specific node's e2e latency is
> similar
> > > to
> > > > > the
> > > > > > task-level e2e latency + task-level processing latency.
> Personally I
> > > > > think
> > > > > > having a task-level record-staleness metric is sufficient.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > > sophie@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > 1. I felt that 50% was not a particularly useful gauge for this
> > > > > specific
> > > > > > > metric, as
> > > > > > > it's presumably most useful at putting an *upper *bound on the
> > > latency
> > > > > you
> > > > > > > can
> > > > > > > reasonably expect to see. I chose percentiles that would
> hopefully
> > > > > give a
> > > > > > > good
> > > > > > > sense of what *most* records will experience, and what *close
> to
> > > all*
> > > > > > > records
> > > > > > > will.
> > > > > > >
> > > > > > > However I'm not married to these specific numbers and could be
> > > > > convinced.
> > > > > > > Would be especially interested in hearing from users on this.
> > > > > > >
> > > > > > > 2. I'm inclined to not include the "hop-to-hop latency" in
> this KIP
> > > > > since
> > > > > > > users
> > > > > > > can always compute it themselves by subtracting the previous
> node's
> > > > > > > end-to-end latency. I guess we could do it either way since
> you can
> > > > > always
> > > > > > > compute one from the other, but I think the end-to-end latency
> > > feels
> > > > > more
> > > > > > > valuable as it's main motivation is not to debug bottlenecks
> in the
> > > > > > > topology but
> > > > > > > to give users a sense of how long it takes arecord to be
> reflected
> > > in
> > > > > > > certain parts
> > > > > > > of the topology. For example this might be useful for users
> who are
> > > > > > > wondering
> > > > > > > roughly when a record that was just produced will be included
> in
> > > their
> > > > > IQ
> > > > > > > results.
> > > > > > > Debugging is just a nice side effect -- but maybe I didn't make
> > > that
> > > > > clear
> > > > > > > enough
> > > > > > > in the KIP's motivation.
> > > > > > >
> > > > > > > 3. Good question, I should address this in the KIP. The short
> > > answer is
> > > > > > > "yes",
> > > > > > > we will include late records. I added a paragraph to the end
> of the
> > > > > > > Proposed
> > > > > > > Changes section explaining the reasoning here, please let me
> know
> > > if
> > > > > you
> > > > > > > have
> > > > > > > any concerns.
> > > > > > >
> > > > > > > 4. Assuming you're referring to the existing metric
> > > "process-latency",
> > > > > that
> > > > > > > metric
> > > > > > > reflects the time for the literal Node#process method to run
> > > whereas
> > > > > this
> > > > > > > metric
> > > > > > > would always be measured relative to the event timestamp.
> > > > > > >
> > > > > > > That said, the naming collision there is pretty confusing so
> I've
> > > > > renamed
> > > > > > > the
> > > > > > > metrics in this KIP to "end-to-end-latency" which I feel better
> > > > > reflects
> > > > > > > the nature
> > > > > > > of the metric anyway.
> > > > > > >
> > > > > > > Thanks for the feedback!
> > > > > > >
> > > > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > > reluctanthero104@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the KIP Sophie. Getting the E2E latency is
> important
> > > for
> > > > > > > > understanding the bottleneck of the application.
> > > > > > > >
> > > > > > > > A couple of questions and ideas:
> > > > > > > >
> > > > > > > > 1. Could you clarify the rational of picking 75, 99 and max
> > > > > percentiles?
> > > > > > > > Normally I see cases where we use 50, 90 percentile as well
> in
> > > > > production
> > > > > > > > systems.
> > > > > > > >
> > > > > > > > 2. The current latency being computed is cumulative, I.E if a
> > > record
> > > > > goes
> > > > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) +
> > > T(A->B) +
> > > > > > > T(A)
> > > > > > > > and so on, where P() represents the captured latency, and T()
> > > > > represents
> > > > > > > > the time for transiting the records between two nodes,
> including
> > > > > > > processing
> > > > > > > > time. For monitoring purpose, maybe having T(B->C) and
> T(A->B)
> > > are
> > > > > more
> > > > > > > > natural to view as "hop-to-hop latency", otherwise if there
> is a
> > > > > spike in
> > > > > > > > T(A->B), both P(B) and P(C) are affected in the same time.
> In the
> > > > > same
> > > > > > > > spirit, the E2E latency is meaningful only when the record
> exits
> > > > > from the
> > > > > > > > sink as this marks the whole time this record spent inside
> the
> > > > > funnel. Do
> > > > > > > > you think we could have separate treatment for sink nodes and
> > > other
> > > > > > > > nodes, so that other nodes only count the time receiving the
> > > record
> > > > > from
> > > > > > > > last hop? I'm not proposing a solution here, just want to
> discuss
> > > > > this
> > > > > > > > alternative to see if it is reasonable.
> > > > > > > >
> > > > > > > > 3. As we are going to monitor late arrival records as well,
> they
> > > > > would
> > > > > > > > create some really spiky graphs when the out-of-order
> records are
> > > > > > > > interleaving with on time records. Should we also supply a
> smooth
> > > > > version
> > > > > > > > of the latency metrics, or user should just take care of it
> by
> > > > > themself?
> > > > > > > >
> > > > > > > > 4. Regarding this new metrics, we haven't discussed its
> relation
> > > > > with our
> > > > > > > > existing processing latency metrics, could you add some
> context
> > > on
> > > > > > > > comparison and a simple `when to use which` tutorial for the
> > > best?
> > > > > > > >
> > > > > > > > Boyang
> > > > > > > >
> > > > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > > > sophie@confluent.io
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey all,
> > > > > > > > >
> > > > > > > > > I'd like to kick off discussion on KIP-613 which aims to
> add
> > > > > end-to-end
> > > > > > > > > latency metrics to Streams. Please take a look:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Sophie
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
Thanks Sophie,

I hope this isn't too nit-picky, but is there a reason to choose "avg" instead
of "mean"? Maybe this is too paranoid, and I might be oversensitive because
of the mistake I just made earlier, but it strikes me that "avg" is actually
ambiguous, as it refers to a family of statistics, whereas "mean" is specific.
I see other Kafka metrics with "avg", but none with "mean"; was that the
reason? If so, I'm +1.

Regarding the names of the percentile, I actually couldn't find _any_ other
metrics that use percentile. Was there a reason to choose "99th" as opposed
to "p99" or any other scheme? This is not a criticism, I'm just primarily asking
for consistency's sake.

Thanks again,
-John

On Wed, May 13, 2020, at 19:19, Sophie Blee-Goldman wrote:
> Alright, I can get behind adding the min metric for the sake of pretty
> graphs
> (and trivial computation).
> 
> I'm still on the fence regarding the mean (or 50th percentile) but I can see
> how users might expect it and find it a bit disorienting not to have. So the
> updated proposed metrics are
> 
> 
>    - record-staleness-max [ms]
>    - record-staleness-99th [ms] *(99th percentile)*
>    - record-staleness-75th [ms] *(75th percentile)*
>    - record-staleness-avg [ms] *(mean)*
>    - record-staleness-min [ms]
> 
> 
> On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org> wrote:
> 
> > Oh boy, I never miss an opportunity to embarrass myself. I guess the mean
> > seems more interesting to me than the median, but neither are as
> > interesting as the higher percentiles (99th and max).
> >
> > Min isn’t really important for any SLAs, but it does round out the mental
> > picture of the distribution. I’ve always graphed min along with the other
> > metrics to help me understand how fast the system can be, which helps in
> > optimization decisions. It’s also a relatively inexpensive metric to
> > compute, so it might be nice to just throw it in.
> >
> > On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > > G1:
> > > I was considering it as the "end-to-end latency *up* to the specific
> > task"
> > > but
> > > I'm happy with "record-staleness" if that drives the point home better.
> > So
> > > it's the
> > > "staleness of the record when it is received by that task" -- will update
> > > the KIP
> > >
> > > B1/J:
> > > I'm struggling to imagine a case where the min would actually be useful,
> > > rather than
> > > just intellectually interesting. I don't feel strongly that we shouldn't
> > > add it, but that's
> > > why I didn't include it from the start. Can you enlighten me with an
> > > example?
> > >
> > > I was also vaguely concerned about the overhead of adding multiple
> > > percentile
> > > metrics. Do we have any data to indicate what kind of performance hit we
> > > take on
> > > metrics computation?
> > >
> > > Also, not to be too pedantic but the 50th percentile would be the median
> > > not the
> > > mean. Would you propose to add the mean *and* the 50th percentile, or
> > just
> > > one
> > > of the two?
> > >
> > > Thanks all!
> > > Sophie
> > >
> > > On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > Hello all, and thanks for the KIP, Sophie,
> > > >
> > > > Just some comments on the discussion so far:
> > > >
> > > > B2/G1:
> > > > In principle, it shouldn't matter whether we report "spans" or
> > > > "end-to-end" latency. But in practice, some of the spans are pretty
> > > > difficult to really measure (like time spent waiting in the topics, or
> > > > time from the event happening to the ETL producer choosing to send it,
> > > > or time spent in send/receive buffers, etc., etc.
> > > >
> > > > In other words, it's practically easier to compute spans by subtracting
> > > > e2e latencies than it is to compute e2e latencies by adding spans. You
> > > > can even just consider that the span computation from e2e always just
> > > > involves subtracting two numbers, whereas computing e2e latency from
> > > > spans involves adding _all_ the spans leading up to the end you care
> > about.
> > > >
> > > > It seems like people really prefer to have spans when they are
> > debugging
> > > > latency problems, whereas e2e latency is a more general measurement
> > > > that basically every person/application cares about and should be
> > > > monitoring.
> > > >
> > > > Altogether, it really seem to provide more value to more people if we
> > > > report
> > > > e2e latencies. Regarding "record-staleness" as a name, I think I have
> > no
> > > > preference, I'd defer to other peoples' intuition.
> > > >
> > > > G2:
> > > > I think the processor-node metric is nice, since the inside of a task
> > can
> > > > introduce a significant amount of latency in some cases. Plus, it's a
> > more
> > > > direct measurement, if you really wanted to know (for the purposes of
> > IQ
> > > > or something) how long it takes source events to "show up" at the
> > store.
> > > >
> > > > I think actually recording it at every processor could be expensive,
> > but we
> > > > already record a bunch of metrics at the node level.
> > > >
> > > > B1:
> > > > I think 50% could be reasonable to record also. Even if it's a poor
> > metric
> > > > for operational purposes, a lot of people might expect to see "mean".
> > > > Actually,
> > > > I was surprised not to see "min". Is there a reason to leave it off?
> > > >
> > > > I might suggest:
> > > > min, mean (50th), 75th, 99th, max
> > > >
> > > > B3:
> > > > I agree we should include late records (though not the ones we drop).
> > > > It may be spiky, but only when there are legitimately some records
> > with a
> > > > high end-to-end latency, which is the whole point of these metrics.
> > > >
> > > > That's it! I don't think I have any other feedback, other than a
> > request to
> > > > also report "min".
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > > >
> > > > > 1) The end-to-end latency includes both the processing latency of the
> > > > task
> > > > > and the latency spent sitting in intermediate topics. I have a
> > similar
> > > > > feeling as Boyang mentioned above that the latency metric of a task A
> > > > > actually measures the latency of the sub-topology up-to but not
> > including
> > > > > the processing of A, which is a bit weird.
> > > > >
> > > > > Maybe the my feeling comes from the name "latency" itself, since
> > today we
> > > > > already have several "latency" metrics already which are measuring
> > > > elapsed
> > > > > system-time for processing a record / etc, while here we are
> > comparing
> > > > the
> > > > > system wallclock time with the record timestamp.
> > > > >
> > > > > Maybe we can consider renaming it as "record-staleness" (note we
> > already
> > > > > have a "record-lateness" metric), in which case recording at the
> > > > > system-time before we start processing the record sounds more
> > natural.
> > > > >
> > > > > 2) With that in mind, I'm wondering if the processor-node-level DEBUG
> > > > > metric is worth to add, given that we already have a task-level
> > > > processing
> > > > > latency metric. Basically, a specific node's e2e latency is similar
> > to
> > > > the
> > > > > task-level e2e latency + task-level processing latency. Personally I
> > > > think
> > > > > having a task-level record-staleness metric is sufficient.
> > > > >
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > > sophie@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > 1. I felt that 50% was not a particularly useful gauge for this
> > > > specific
> > > > > > metric, as
> > > > > > it's presumably most useful at putting an *upper *bound on the
> > latency
> > > > you
> > > > > > can
> > > > > > reasonably expect to see. I chose percentiles that would hopefully
> > > > give a
> > > > > > good
> > > > > > sense of what *most* records will experience, and what *close to
> > all*
> > > > > > records
> > > > > > will.
> > > > > >
> > > > > > However I'm not married to these specific numbers and could be
> > > > convinced.
> > > > > > Would be especially interested in hearing from users on this.
> > > > > >
> > > > > > 2. I'm inclined to not include the "hop-to-hop latency" in this KIP
> > > > since
> > > > > > users
> > > > > > can always compute it themselves by subtracting the previous node's
> > > > > > end-to-end latency. I guess we could do it either way since you can
> > > > always
> > > > > > compute one from the other, but I think the end-to-end latency
> > feels
> > > > more
> > > > > > valuable as it's main motivation is not to debug bottlenecks in the
> > > > > > topology but
> > > > > > to give users a sense of how long it takes arecord to be reflected
> > in
> > > > > > certain parts
> > > > > > of the topology. For example this might be useful for users who are
> > > > > > wondering
> > > > > > roughly when a record that was just produced will be included in
> > their
> > > > IQ
> > > > > > results.
> > > > > > Debugging is just a nice side effect -- but maybe I didn't make
> > that
> > > > clear
> > > > > > enough
> > > > > > in the KIP's motivation.
> > > > > >
> > > > > > 3. Good question, I should address this in the KIP. The short
> > answer is
> > > > > > "yes",
> > > > > > we will include late records. I added a paragraph to the end of the
> > > > > > Proposed
> > > > > > Changes section explaining the reasoning here, please let me know
> > if
> > > > you
> > > > > > have
> > > > > > any concerns.
> > > > > >
> > > > > > 4. Assuming you're referring to the existing metric
> > "process-latency",
> > > > that
> > > > > > metric
> > > > > > reflects the time for the literal Node#process method to run
> > whereas
> > > > this
> > > > > > metric
> > > > > > would always be measured relative to the event timestamp.
> > > > > >
> > > > > > That said, the naming collision there is pretty confusing so I've
> > > > renamed
> > > > > > the
> > > > > > metrics in this KIP to "end-to-end-latency" which I feel better
> > > > reflects
> > > > > > the nature
> > > > > > of the metric anyway.
> > > > > >
> > > > > > Thanks for the feedback!
> > > > > >
> > > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > > reluctanthero104@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the KIP Sophie. Getting the E2E latency is important
> > for
> > > > > > > understanding the bottleneck of the application.
> > > > > > >
> > > > > > > A couple of questions and ideas:
> > > > > > >
> > > > > > > 1. Could you clarify the rational of picking 75, 99 and max
> > > > percentiles?
> > > > > > > Normally I see cases where we use 50, 90 percentile as well in
> > > > production
> > > > > > > systems.
> > > > > > >
> > > > > > > 2. The current latency being computed is cumulative, I.E if a
> > record
> > > > goes
> > > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) +
> > T(A->B) +
> > > > > > T(A)
> > > > > > > and so on, where P() represents the captured latency, and T()
> > > > represents
> > > > > > > the time for transiting the records between two nodes, including
> > > > > > processing
> > > > > > > time. For monitoring purpose, maybe having T(B->C) and T(A->B)
> > are
> > > > more
> > > > > > > natural to view as "hop-to-hop latency", otherwise if there is a
> > > > spike in
> > > > > > > T(A->B), both P(B) and P(C) are affected in the same time. In the
> > > > same
> > > > > > > spirit, the E2E latency is meaningful only when the record exits
> > > > from the
> > > > > > > sink as this marks the whole time this record spent inside the
> > > > funnel. Do
> > > > > > > you think we could have separate treatment for sink nodes and
> > other
> > > > > > > nodes, so that other nodes only count the time receiving the
> > record
> > > > from
> > > > > > > last hop? I'm not proposing a solution here, just want to discuss
> > > > this
> > > > > > > alternative to see if it is reasonable.
> > > > > > >
> > > > > > > 3. As we are going to monitor late arrival records as well, they
> > > > would
> > > > > > > create some really spiky graphs when the out-of-order records are
> > > > > > > interleaving with on time records. Should we also supply a smooth
> > > > version
> > > > > > > of the latency metrics, or user should just take care of it by
> > > > themself?
> > > > > > >
> > > > > > > 4. Regarding this new metrics, we haven't discussed its relation
> > > > with our
> > > > > > > existing processing latency metrics, could you add some context
> > on
> > > > > > > comparison and a simple `when to use which` tutorial for the
> > best?
> > > > > > >
> > > > > > > Boyang
> > > > > > >
> > > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > > sophie@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > I'd like to kick off discussion on KIP-613 which aims to add
> > > > end-to-end
> > > > > > > > latency metrics to Streams. Please take a look:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Sophie
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
Alright, I can get behind adding the min metric for the sake of pretty
graphs
(and trivial computation).

I'm still on the fence regarding the mean (or 50th percentile) but I can see
how users might expect it and find it a bit disorienting not to have. So the
updated proposed metrics are


   - record-staleness-max [ms]
   - record-staleness-99th [ms] *(99th percentile)*
   - record-staleness-75th [ms] *(75th percentile)*
   - record-staleness-avg [ms] *(mean)*
   - record-staleness-min [ms]


On Wed, May 13, 2020 at 4:42 PM John Roesler <vv...@apache.org> wrote:

> Oh boy, I never miss an opportunity to embarrass myself. I guess the mean
> seems more interesting to me than the median, but neither are as
> interesting as the higher percentiles (99th and max).
>
> Min isn’t really important for any SLAs, but it does round out the mental
> picture of the distribution. I’ve always graphed min along with the other
> metrics to help me understand how fast the system can be, which helps in
> optimization decisions. It’s also a relatively inexpensive metric to
> compute, so it might be nice to just throw it in.
>
> On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> > G1:
> > I was considering it as the "end-to-end latency *up* to the specific
> task"
> > but
> > I'm happy with "record-staleness" if that drives the point home better.
> So
> > it's the
> > "staleness of the record when it is received by that task" -- will update
> > the KIP
> >
> > B1/J:
> > I'm struggling to imagine a case where the min would actually be useful,
> > rather than
> > just intellectually interesting. I don't feel strongly that we shouldn't
> > add it, but that's
> > why I didn't include it from the start. Can you enlighten me with an
> > example?
> >
> > I was also vaguely concerned about the overhead of adding multiple
> > percentile
> > metrics. Do we have any data to indicate what kind of performance hit we
> > take on
> > metrics computation?
> >
> > Also, not to be too pedantic but the 50th percentile would be the median
> > not the
> > mean. Would you propose to add the mean *and* the 50th percentile, or
> just
> > one
> > of the two?
> >
> > Thanks all!
> > Sophie
> >
> > On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org>
> wrote:
> >
> > > Hello all, and thanks for the KIP, Sophie,
> > >
> > > Just some comments on the discussion so far:
> > >
> > > B2/G1:
> > > In principle, it shouldn't matter whether we report "spans" or
> > > "end-to-end" latency. But in practice, some of the spans are pretty
> > > difficult to really measure (like time spent waiting in the topics, or
> > > time from the event happening to the ETL producer choosing to send it,
> > > or time spent in send/receive buffers, etc., etc.
> > >
> > > In other words, it's practically easier to compute spans by subtracting
> > > e2e latencies than it is to compute e2e latencies by adding spans. You
> > > can even just consider that the span computation from e2e always just
> > > involves subtracting two numbers, whereas computing e2e latency from
> > > spans involves adding _all_ the spans leading up to the end you care
> about.
> > >
> > > It seems like people really prefer to have spans when they are
> debugging
> > > latency problems, whereas e2e latency is a more general measurement
> > > that basically every person/application cares about and should be
> > > monitoring.
> > >
> > > Altogether, it really seem to provide more value to more people if we
> > > report
> > > e2e latencies. Regarding "record-staleness" as a name, I think I have
> no
> > > preference, I'd defer to other peoples' intuition.
> > >
> > > G2:
> > > I think the processor-node metric is nice, since the inside of a task
> can
> > > introduce a significant amount of latency in some cases. Plus, it's a
> more
> > > direct measurement, if you really wanted to know (for the purposes of
> IQ
> > > or something) how long it takes source events to "show up" at the
> store.
> > >
> > > I think actually recording it at every processor could be expensive,
> but we
> > > already record a bunch of metrics at the node level.
> > >
> > > B1:
> > > I think 50% could be reasonable to record also. Even if it's a poor
> metric
> > > for operational purposes, a lot of people might expect to see "mean".
> > > Actually,
> > > I was surprised not to see "min". Is there a reason to leave it off?
> > >
> > > I might suggest:
> > > min, mean (50th), 75th, 99th, max
> > >
> > > B3:
> > > I agree we should include late records (though not the ones we drop).
> > > It may be spiky, but only when there are legitimately some records
> with a
> > > high end-to-end latency, which is the whole point of these metrics.
> > >
> > > That's it! I don't think I have any other feedback, other than a
> request to
> > > also report "min".
> > >
> > > Thanks,
> > > -John
> > >
> > > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > > Thanks Sophie for the KIP, a few quick thoughts:
> > > >
> > > > 1) The end-to-end latency includes both the processing latency of the
> > > task
> > > > and the latency spent sitting in intermediate topics. I have a
> similar
> > > > feeling as Boyang mentioned above that the latency metric of a task A
> > > > actually measures the latency of the sub-topology up-to but not
> including
> > > > the processing of A, which is a bit weird.
> > > >
> > > > Maybe the my feeling comes from the name "latency" itself, since
> today we
> > > > already have several "latency" metrics already which are measuring
> > > elapsed
> > > > system-time for processing a record / etc, while here we are
> comparing
> > > the
> > > > system wallclock time with the record timestamp.
> > > >
> > > > Maybe we can consider renaming it as "record-staleness" (note we
> already
> > > > have a "record-lateness" metric), in which case recording at the
> > > > system-time before we start processing the record sounds more
> natural.
> > > >
> > > > 2) With that in mind, I'm wondering if the processor-node-level DEBUG
> > > > metric is worth to add, given that we already have a task-level
> > > processing
> > > > latency metric. Basically, a specific node's e2e latency is similar
> to
> > > the
> > > > task-level e2e latency + task-level processing latency. Personally I
> > > think
> > > > having a task-level record-staleness metric is sufficient.
> > > >
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > > sophie@confluent.io>
> > > > wrote:
> > > >
> > > > > 1. I felt that 50% was not a particularly useful gauge for this
> > > specific
> > > > > metric, as
> > > > > it's presumably most useful at putting an *upper *bound on the
> latency
> > > you
> > > > > can
> > > > > reasonably expect to see. I chose percentiles that would hopefully
> > > give a
> > > > > good
> > > > > sense of what *most* records will experience, and what *close to
> all*
> > > > > records
> > > > > will.
> > > > >
> > > > > However I'm not married to these specific numbers and could be
> > > convinced.
> > > > > Would be especially interested in hearing from users on this.
> > > > >
> > > > > 2. I'm inclined to not include the "hop-to-hop latency" in this KIP
> > > since
> > > > > users
> > > > > can always compute it themselves by subtracting the previous node's
> > > > > end-to-end latency. I guess we could do it either way since you can
> > > always
> > > > > compute one from the other, but I think the end-to-end latency
> feels
> > > more
> > > > > valuable as it's main motivation is not to debug bottlenecks in the
> > > > > topology but
> > > > > to give users a sense of how long it takes arecord to be reflected
> in
> > > > > certain parts
> > > > > of the topology. For example this might be useful for users who are
> > > > > wondering
> > > > > roughly when a record that was just produced will be included in
> their
> > > IQ
> > > > > results.
> > > > > Debugging is just a nice side effect -- but maybe I didn't make
> that
> > > clear
> > > > > enough
> > > > > in the KIP's motivation.
> > > > >
> > > > > 3. Good question, I should address this in the KIP. The short
> answer is
> > > > > "yes",
> > > > > we will include late records. I added a paragraph to the end of the
> > > > > Proposed
> > > > > Changes section explaining the reasoning here, please let me know
> if
> > > you
> > > > > have
> > > > > any concerns.
> > > > >
> > > > > 4. Assuming you're referring to the existing metric
> "process-latency",
> > > that
> > > > > metric
> > > > > reflects the time for the literal Node#process method to run
> whereas
> > > this
> > > > > metric
> > > > > would always be measured relative to the event timestamp.
> > > > >
> > > > > That said, the naming collision there is pretty confusing so I've
> > > renamed
> > > > > the
> > > > > metrics in this KIP to "end-to-end-latency" which I feel better
> > > reflects
> > > > > the nature
> > > > > of the metric anyway.
> > > > >
> > > > > Thanks for the feedback!
> > > > >
> > > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > > reluctanthero104@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the KIP Sophie. Getting the E2E latency is important
> for
> > > > > > understanding the bottleneck of the application.
> > > > > >
> > > > > > A couple of questions and ideas:
> > > > > >
> > > > > > 1. Could you clarify the rational of picking 75, 99 and max
> > > percentiles?
> > > > > > Normally I see cases where we use 50, 90 percentile as well in
> > > production
> > > > > > systems.
> > > > > >
> > > > > > 2. The current latency being computed is cumulative, I.E if a
> record
> > > goes
> > > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) +
> T(A->B) +
> > > > > T(A)
> > > > > > and so on, where P() represents the captured latency, and T()
> > > represents
> > > > > > the time for transiting the records between two nodes, including
> > > > > processing
> > > > > > time. For monitoring purpose, maybe having T(B->C) and T(A->B)
> are
> > > more
> > > > > > natural to view as "hop-to-hop latency", otherwise if there is a
> > > spike in
> > > > > > T(A->B), both P(B) and P(C) are affected in the same time. In the
> > > same
> > > > > > spirit, the E2E latency is meaningful only when the record exits
> > > from the
> > > > > > sink as this marks the whole time this record spent inside the
> > > funnel. Do
> > > > > > you think we could have separate treatment for sink nodes and
> other
> > > > > > nodes, so that other nodes only count the time receiving the
> record
> > > from
> > > > > > last hop? I'm not proposing a solution here, just want to discuss
> > > this
> > > > > > alternative to see if it is reasonable.
> > > > > >
> > > > > > 3. As we are going to monitor late arrival records as well, they
> > > would
> > > > > > create some really spiky graphs when the out-of-order records are
> > > > > > interleaving with on time records. Should we also supply a smooth
> > > version
> > > > > > of the latency metrics, or user should just take care of it by
> > > themself?
> > > > > >
> > > > > > 4. Regarding this new metrics, we haven't discussed its relation
> > > with our
> > > > > > existing processing latency metrics, could you add some context
> on
> > > > > > comparison and a simple `when to use which` tutorial for the
> best?
> > > > > >
> > > > > > Boyang
> > > > > >
> > > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > > sophie@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > I'd like to kick off discussion on KIP-613 which aims to add
> > > end-to-end
> > > > > > > latency metrics to Streams. Please take a look:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Sophie
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
Oh boy, I never miss an opportunity to embarrass myself. I guess the mean seems more interesting to me than the median, but neither are as interesting as the higher percentiles (99th and max). 

Min isn’t really important for any SLAs, but it does round out the mental picture of the distribution. I’ve always graphed min along with the other metrics to help me understand how fast the system can be, which helps in optimization decisions. It’s also a relatively inexpensive metric to compute, so it might be nice to just throw it in. 

On Wed, May 13, 2020, at 18:18, Sophie Blee-Goldman wrote:
> G1:
> I was considering it as the "end-to-end latency *up* to the specific task"
> but
> I'm happy with "record-staleness" if that drives the point home better. So
> it's the
> "staleness of the record when it is received by that task" -- will update
> the KIP
> 
> B1/J:
> I'm struggling to imagine a case where the min would actually be useful,
> rather than
> just intellectually interesting. I don't feel strongly that we shouldn't
> add it, but that's
> why I didn't include it from the start. Can you enlighten me with an
> example?
> 
> I was also vaguely concerned about the overhead of adding multiple
> percentile
> metrics. Do we have any data to indicate what kind of performance hit we
> take on
> metrics computation?
> 
> Also, not to be too pedantic but the 50th percentile would be the median
> not the
> mean. Would you propose to add the mean *and* the 50th percentile, or just
> one
> of the two?
> 
> Thanks all!
> Sophie
> 
> On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org> wrote:
> 
> > Hello all, and thanks for the KIP, Sophie,
> >
> > Just some comments on the discussion so far:
> >
> > B2/G1:
> > In principle, it shouldn't matter whether we report "spans" or
> > "end-to-end" latency. But in practice, some of the spans are pretty
> > difficult to really measure (like time spent waiting in the topics, or
> > time from the event happening to the ETL producer choosing to send it,
> > or time spent in send/receive buffers, etc., etc.
> >
> > In other words, it's practically easier to compute spans by subtracting
> > e2e latencies than it is to compute e2e latencies by adding spans. You
> > can even just consider that the span computation from e2e always just
> > involves subtracting two numbers, whereas computing e2e latency from
> > spans involves adding _all_ the spans leading up to the end you care about.
> >
> > It seems like people really prefer to have spans when they are debugging
> > latency problems, whereas e2e latency is a more general measurement
> > that basically every person/application cares about and should be
> > monitoring.
> >
> > Altogether, it really seem to provide more value to more people if we
> > report
> > e2e latencies. Regarding "record-staleness" as a name, I think I have no
> > preference, I'd defer to other peoples' intuition.
> >
> > G2:
> > I think the processor-node metric is nice, since the inside of a task can
> > introduce a significant amount of latency in some cases. Plus, it's a more
> > direct measurement, if you really wanted to know (for the purposes of IQ
> > or something) how long it takes source events to "show up" at the store.
> >
> > I think actually recording it at every processor could be expensive, but we
> > already record a bunch of metrics at the node level.
> >
> > B1:
> > I think 50% could be reasonable to record also. Even if it's a poor metric
> > for operational purposes, a lot of people might expect to see "mean".
> > Actually,
> > I was surprised not to see "min". Is there a reason to leave it off?
> >
> > I might suggest:
> > min, mean (50th), 75th, 99th, max
> >
> > B3:
> > I agree we should include late records (though not the ones we drop).
> > It may be spiky, but only when there are legitimately some records with a
> > high end-to-end latency, which is the whole point of these metrics.
> >
> > That's it! I don't think I have any other feedback, other than a request to
> > also report "min".
> >
> > Thanks,
> > -John
> >
> > On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > > Thanks Sophie for the KIP, a few quick thoughts:
> > >
> > > 1) The end-to-end latency includes both the processing latency of the
> > task
> > > and the latency spent sitting in intermediate topics. I have a similar
> > > feeling as Boyang mentioned above that the latency metric of a task A
> > > actually measures the latency of the sub-topology up-to but not including
> > > the processing of A, which is a bit weird.
> > >
> > > Maybe the my feeling comes from the name "latency" itself, since today we
> > > already have several "latency" metrics already which are measuring
> > elapsed
> > > system-time for processing a record / etc, while here we are comparing
> > the
> > > system wallclock time with the record timestamp.
> > >
> > > Maybe we can consider renaming it as "record-staleness" (note we already
> > > have a "record-lateness" metric), in which case recording at the
> > > system-time before we start processing the record sounds more natural.
> > >
> > > 2) With that in mind, I'm wondering if the processor-node-level DEBUG
> > > metric is worth to add, given that we already have a task-level
> > processing
> > > latency metric. Basically, a specific node's e2e latency is similar to
> > the
> > > task-level e2e latency + task-level processing latency. Personally I
> > think
> > > having a task-level record-staleness metric is sufficient.
> > >
> > >
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> > sophie@confluent.io>
> > > wrote:
> > >
> > > > 1. I felt that 50% was not a particularly useful gauge for this
> > specific
> > > > metric, as
> > > > it's presumably most useful at putting an *upper *bound on the latency
> > you
> > > > can
> > > > reasonably expect to see. I chose percentiles that would hopefully
> > give a
> > > > good
> > > > sense of what *most* records will experience, and what *close to all*
> > > > records
> > > > will.
> > > >
> > > > However I'm not married to these specific numbers and could be
> > convinced.
> > > > Would be especially interested in hearing from users on this.
> > > >
> > > > 2. I'm inclined to not include the "hop-to-hop latency" in this KIP
> > since
> > > > users
> > > > can always compute it themselves by subtracting the previous node's
> > > > end-to-end latency. I guess we could do it either way since you can
> > always
> > > > compute one from the other, but I think the end-to-end latency feels
> > more
> > > > valuable as it's main motivation is not to debug bottlenecks in the
> > > > topology but
> > > > to give users a sense of how long it takes arecord to be reflected in
> > > > certain parts
> > > > of the topology. For example this might be useful for users who are
> > > > wondering
> > > > roughly when a record that was just produced will be included in their
> > IQ
> > > > results.
> > > > Debugging is just a nice side effect -- but maybe I didn't make that
> > clear
> > > > enough
> > > > in the KIP's motivation.
> > > >
> > > > 3. Good question, I should address this in the KIP. The short answer is
> > > > "yes",
> > > > we will include late records. I added a paragraph to the end of the
> > > > Proposed
> > > > Changes section explaining the reasoning here, please let me know if
> > you
> > > > have
> > > > any concerns.
> > > >
> > > > 4. Assuming you're referring to the existing metric "process-latency",
> > that
> > > > metric
> > > > reflects the time for the literal Node#process method to run whereas
> > this
> > > > metric
> > > > would always be measured relative to the event timestamp.
> > > >
> > > > That said, the naming collision there is pretty confusing so I've
> > renamed
> > > > the
> > > > metrics in this KIP to "end-to-end-latency" which I feel better
> > reflects
> > > > the nature
> > > > of the metric anyway.
> > > >
> > > > Thanks for the feedback!
> > > >
> > > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> > reluctanthero104@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for the KIP Sophie. Getting the E2E latency is important for
> > > > > understanding the bottleneck of the application.
> > > > >
> > > > > A couple of questions and ideas:
> > > > >
> > > > > 1. Could you clarify the rational of picking 75, 99 and max
> > percentiles?
> > > > > Normally I see cases where we use 50, 90 percentile as well in
> > production
> > > > > systems.
> > > > >
> > > > > 2. The current latency being computed is cumulative, I.E if a record
> > goes
> > > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) +
> > > > T(A)
> > > > > and so on, where P() represents the captured latency, and T()
> > represents
> > > > > the time for transiting the records between two nodes, including
> > > > processing
> > > > > time. For monitoring purpose, maybe having T(B->C) and T(A->B) are
> > more
> > > > > natural to view as "hop-to-hop latency", otherwise if there is a
> > spike in
> > > > > T(A->B), both P(B) and P(C) are affected in the same time. In the
> > same
> > > > > spirit, the E2E latency is meaningful only when the record exits
> > from the
> > > > > sink as this marks the whole time this record spent inside the
> > funnel. Do
> > > > > you think we could have separate treatment for sink nodes and other
> > > > > nodes, so that other nodes only count the time receiving the record
> > from
> > > > > last hop? I'm not proposing a solution here, just want to discuss
> > this
> > > > > alternative to see if it is reasonable.
> > > > >
> > > > > 3. As we are going to monitor late arrival records as well, they
> > would
> > > > > create some really spiky graphs when the out-of-order records are
> > > > > interleaving with on time records. Should we also supply a smooth
> > version
> > > > > of the latency metrics, or user should just take care of it by
> > themself?
> > > > >
> > > > > 4. Regarding this new metrics, we haven't discussed its relation
> > with our
> > > > > existing processing latency metrics, could you add some context on
> > > > > comparison and a simple `when to use which` tutorial for the best?
> > > > >
> > > > > Boyang
> > > > >
> > > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> > sophie@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I'd like to kick off discussion on KIP-613 which aims to add
> > end-to-end
> > > > > > latency metrics to Streams. Please take a look:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > > >
> > > > > > Cheers,
> > > > > > Sophie
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
G1:
I was considering it as the "end-to-end latency *up* to the specific task"
but
I'm happy with "record-staleness" if that drives the point home better. So
it's the
"staleness of the record when it is received by that task" -- will update
the KIP

B1/J:
I'm struggling to imagine a case where the min would actually be useful,
rather than
just intellectually interesting. I don't feel strongly that we shouldn't
add it, but that's
why I didn't include it from the start. Can you enlighten me with an
example?

I was also vaguely concerned about the overhead of adding multiple
percentile
metrics. Do we have any data to indicate what kind of performance hit we
take on
metrics computation?

Also, not to be too pedantic but the 50th percentile would be the median
not the
mean. Would you propose to add the mean *and* the 50th percentile, or just
one
of the two?

Thanks all!
Sophie

On Wed, May 13, 2020 at 3:34 PM John Roesler <vv...@apache.org> wrote:

> Hello all, and thanks for the KIP, Sophie,
>
> Just some comments on the discussion so far:
>
> B2/G1:
> In principle, it shouldn't matter whether we report "spans" or
> "end-to-end" latency. But in practice, some of the spans are pretty
> difficult to really measure (like time spent waiting in the topics, or
> time from the event happening to the ETL producer choosing to send it,
> or time spent in send/receive buffers, etc., etc.
>
> In other words, it's practically easier to compute spans by subtracting
> e2e latencies than it is to compute e2e latencies by adding spans. You
> can even just consider that the span computation from e2e always just
> involves subtracting two numbers, whereas computing e2e latency from
> spans involves adding _all_ the spans leading up to the end you care about.
>
> It seems like people really prefer to have spans when they are debugging
> latency problems, whereas e2e latency is a more general measurement
> that basically every person/application cares about and should be
> monitoring.
>
> Altogether, it really seem to provide more value to more people if we
> report
> e2e latencies. Regarding "record-staleness" as a name, I think I have no
> preference, I'd defer to other peoples' intuition.
>
> G2:
> I think the processor-node metric is nice, since the inside of a task can
> introduce a significant amount of latency in some cases. Plus, it's a more
> direct measurement, if you really wanted to know (for the purposes of IQ
> or something) how long it takes source events to "show up" at the store.
>
> I think actually recording it at every processor could be expensive, but we
> already record a bunch of metrics at the node level.
>
> B1:
> I think 50% could be reasonable to record also. Even if it's a poor metric
> for operational purposes, a lot of people might expect to see "mean".
> Actually,
> I was surprised not to see "min". Is there a reason to leave it off?
>
> I might suggest:
> min, mean (50th), 75th, 99th, max
>
> B3:
> I agree we should include late records (though not the ones we drop).
> It may be spiky, but only when there are legitimately some records with a
> high end-to-end latency, which is the whole point of these metrics.
>
> That's it! I don't think I have any other feedback, other than a request to
> also report "min".
>
> Thanks,
> -John
>
> On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> > Thanks Sophie for the KIP, a few quick thoughts:
> >
> > 1) The end-to-end latency includes both the processing latency of the
> task
> > and the latency spent sitting in intermediate topics. I have a similar
> > feeling as Boyang mentioned above that the latency metric of a task A
> > actually measures the latency of the sub-topology up-to but not including
> > the processing of A, which is a bit weird.
> >
> > Maybe the my feeling comes from the name "latency" itself, since today we
> > already have several "latency" metrics already which are measuring
> elapsed
> > system-time for processing a record / etc, while here we are comparing
> the
> > system wallclock time with the record timestamp.
> >
> > Maybe we can consider renaming it as "record-staleness" (note we already
> > have a "record-lateness" metric), in which case recording at the
> > system-time before we start processing the record sounds more natural.
> >
> > 2) With that in mind, I'm wondering if the processor-node-level DEBUG
> > metric is worth to add, given that we already have a task-level
> processing
> > latency metric. Basically, a specific node's e2e latency is similar to
> the
> > task-level e2e latency + task-level processing latency. Personally I
> think
> > having a task-level record-staleness metric is sufficient.
> >
> >
> >
> > Guozhang
> >
> >
> >
> > On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <
> sophie@confluent.io>
> > wrote:
> >
> > > 1. I felt that 50% was not a particularly useful gauge for this
> specific
> > > metric, as
> > > it's presumably most useful at putting an *upper *bound on the latency
> you
> > > can
> > > reasonably expect to see. I chose percentiles that would hopefully
> give a
> > > good
> > > sense of what *most* records will experience, and what *close to all*
> > > records
> > > will.
> > >
> > > However I'm not married to these specific numbers and could be
> convinced.
> > > Would be especially interested in hearing from users on this.
> > >
> > > 2. I'm inclined to not include the "hop-to-hop latency" in this KIP
> since
> > > users
> > > can always compute it themselves by subtracting the previous node's
> > > end-to-end latency. I guess we could do it either way since you can
> always
> > > compute one from the other, but I think the end-to-end latency feels
> more
> > > valuable as it's main motivation is not to debug bottlenecks in the
> > > topology but
> > > to give users a sense of how long it takes arecord to be reflected in
> > > certain parts
> > > of the topology. For example this might be useful for users who are
> > > wondering
> > > roughly when a record that was just produced will be included in their
> IQ
> > > results.
> > > Debugging is just a nice side effect -- but maybe I didn't make that
> clear
> > > enough
> > > in the KIP's motivation.
> > >
> > > 3. Good question, I should address this in the KIP. The short answer is
> > > "yes",
> > > we will include late records. I added a paragraph to the end of the
> > > Proposed
> > > Changes section explaining the reasoning here, please let me know if
> you
> > > have
> > > any concerns.
> > >
> > > 4. Assuming you're referring to the existing metric "process-latency",
> that
> > > metric
> > > reflects the time for the literal Node#process method to run whereas
> this
> > > metric
> > > would always be measured relative to the event timestamp.
> > >
> > > That said, the naming collision there is pretty confusing so I've
> renamed
> > > the
> > > metrics in this KIP to "end-to-end-latency" which I feel better
> reflects
> > > the nature
> > > of the metric anyway.
> > >
> > > Thanks for the feedback!
> > >
> > > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <
> reluctanthero104@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the KIP Sophie. Getting the E2E latency is important for
> > > > understanding the bottleneck of the application.
> > > >
> > > > A couple of questions and ideas:
> > > >
> > > > 1. Could you clarify the rational of picking 75, 99 and max
> percentiles?
> > > > Normally I see cases where we use 50, 90 percentile as well in
> production
> > > > systems.
> > > >
> > > > 2. The current latency being computed is cumulative, I.E if a record
> goes
> > > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) +
> > > T(A)
> > > > and so on, where P() represents the captured latency, and T()
> represents
> > > > the time for transiting the records between two nodes, including
> > > processing
> > > > time. For monitoring purpose, maybe having T(B->C) and T(A->B) are
> more
> > > > natural to view as "hop-to-hop latency", otherwise if there is a
> spike in
> > > > T(A->B), both P(B) and P(C) are affected in the same time. In the
> same
> > > > spirit, the E2E latency is meaningful only when the record exits
> from the
> > > > sink as this marks the whole time this record spent inside the
> funnel. Do
> > > > you think we could have separate treatment for sink nodes and other
> > > > nodes, so that other nodes only count the time receiving the record
> from
> > > > last hop? I'm not proposing a solution here, just want to discuss
> this
> > > > alternative to see if it is reasonable.
> > > >
> > > > 3. As we are going to monitor late arrival records as well, they
> would
> > > > create some really spiky graphs when the out-of-order records are
> > > > interleaving with on time records. Should we also supply a smooth
> version
> > > > of the latency metrics, or user should just take care of it by
> themself?
> > > >
> > > > 4. Regarding this new metrics, we haven't discussed its relation
> with our
> > > > existing processing latency metrics, could you add some context on
> > > > comparison and a simple `when to use which` tutorial for the best?
> > > >
> > > > Boyang
> > > >
> > > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <
> sophie@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > I'd like to kick off discussion on KIP-613 which aims to add
> end-to-end
> > > > > latency metrics to Streams. Please take a look:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > > >
> > > > > Cheers,
> > > > > Sophie
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by John Roesler <vv...@apache.org>.
Hello all, and thanks for the KIP, Sophie,

Just some comments on the discussion so far:

B2/G1:
In principle, it shouldn't matter whether we report "spans" or
"end-to-end" latency. But in practice, some of the spans are pretty
difficult to really measure (like time spent waiting in the topics, or 
time from the event happening to the ETL producer choosing to send it,
or time spent in send/receive buffers, etc., etc.

In other words, it's practically easier to compute spans by subtracting
e2e latencies than it is to compute e2e latencies by adding spans. You
can even just consider that the span computation from e2e always just
involves subtracting two numbers, whereas computing e2e latency from
spans involves adding _all_ the spans leading up to the end you care about.

It seems like people really prefer to have spans when they are debugging
latency problems, whereas e2e latency is a more general measurement
that basically every person/application cares about and should be
monitoring.

Altogether, it really seem to provide more value to more people if we report
e2e latencies. Regarding "record-staleness" as a name, I think I have no
preference, I'd defer to other peoples' intuition.

G2:
I think the processor-node metric is nice, since the inside of a task can
introduce a significant amount of latency in some cases. Plus, it's a more
direct measurement, if you really wanted to know (for the purposes of IQ
or something) how long it takes source events to "show up" at the store.

I think actually recording it at every processor could be expensive, but we
already record a bunch of metrics at the node level.

B1:
I think 50% could be reasonable to record also. Even if it's a poor metric
for operational purposes, a lot of people might expect to see "mean". Actually,
I was surprised not to see "min". Is there a reason to leave it off?

I might suggest:
min, mean (50th), 75th, 99th, max

B3:
I agree we should include late records (though not the ones we drop).
It may be spiky, but only when there are legitimately some records with a
high end-to-end latency, which is the whole point of these metrics.

That's it! I don't think I have any other feedback, other than a request to
also report "min".

Thanks,
-John

On Wed, May 13, 2020, at 16:58, Guozhang Wang wrote:
> Thanks Sophie for the KIP, a few quick thoughts:
> 
> 1) The end-to-end latency includes both the processing latency of the task
> and the latency spent sitting in intermediate topics. I have a similar
> feeling as Boyang mentioned above that the latency metric of a task A
> actually measures the latency of the sub-topology up-to but not including
> the processing of A, which is a bit weird.
> 
> Maybe the my feeling comes from the name "latency" itself, since today we
> already have several "latency" metrics already which are measuring elapsed
> system-time for processing a record / etc, while here we are comparing the
> system wallclock time with the record timestamp.
> 
> Maybe we can consider renaming it as "record-staleness" (note we already
> have a "record-lateness" metric), in which case recording at the
> system-time before we start processing the record sounds more natural.
> 
> 2) With that in mind, I'm wondering if the processor-node-level DEBUG
> metric is worth to add, given that we already have a task-level processing
> latency metric. Basically, a specific node's e2e latency is similar to the
> task-level e2e latency + task-level processing latency. Personally I think
> having a task-level record-staleness metric is sufficient.
> 
> 
> 
> Guozhang
> 
> 
> 
> On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
> 
> > 1. I felt that 50% was not a particularly useful gauge for this specific
> > metric, as
> > it's presumably most useful at putting an *upper *bound on the latency you
> > can
> > reasonably expect to see. I chose percentiles that would hopefully give a
> > good
> > sense of what *most* records will experience, and what *close to all*
> > records
> > will.
> >
> > However I'm not married to these specific numbers and could be convinced.
> > Would be especially interested in hearing from users on this.
> >
> > 2. I'm inclined to not include the "hop-to-hop latency" in this KIP since
> > users
> > can always compute it themselves by subtracting the previous node's
> > end-to-end latency. I guess we could do it either way since you can always
> > compute one from the other, but I think the end-to-end latency feels more
> > valuable as it's main motivation is not to debug bottlenecks in the
> > topology but
> > to give users a sense of how long it takes arecord to be reflected in
> > certain parts
> > of the topology. For example this might be useful for users who are
> > wondering
> > roughly when a record that was just produced will be included in their IQ
> > results.
> > Debugging is just a nice side effect -- but maybe I didn't make that clear
> > enough
> > in the KIP's motivation.
> >
> > 3. Good question, I should address this in the KIP. The short answer is
> > "yes",
> > we will include late records. I added a paragraph to the end of the
> > Proposed
> > Changes section explaining the reasoning here, please let me know if you
> > have
> > any concerns.
> >
> > 4. Assuming you're referring to the existing metric "process-latency", that
> > metric
> > reflects the time for the literal Node#process method to run whereas this
> > metric
> > would always be measured relative to the event timestamp.
> >
> > That said, the naming collision there is pretty confusing so I've renamed
> > the
> > metrics in this KIP to "end-to-end-latency" which I feel better reflects
> > the nature
> > of the metric anyway.
> >
> > Thanks for the feedback!
> >
> > On Wed, May 13, 2020 at 10:21 AM Boyang Chen <re...@gmail.com>
> > wrote:
> >
> > > Thanks for the KIP Sophie. Getting the E2E latency is important for
> > > understanding the bottleneck of the application.
> > >
> > > A couple of questions and ideas:
> > >
> > > 1. Could you clarify the rational of picking 75, 99 and max percentiles?
> > > Normally I see cases where we use 50, 90 percentile as well in production
> > > systems.
> > >
> > > 2. The current latency being computed is cumulative, I.E if a record goes
> > > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) +
> > T(A)
> > > and so on, where P() represents the captured latency, and T() represents
> > > the time for transiting the records between two nodes, including
> > processing
> > > time. For monitoring purpose, maybe having T(B->C) and T(A->B) are more
> > > natural to view as "hop-to-hop latency", otherwise if there is a spike in
> > > T(A->B), both P(B) and P(C) are affected in the same time. In the same
> > > spirit, the E2E latency is meaningful only when the record exits from the
> > > sink as this marks the whole time this record spent inside the funnel. Do
> > > you think we could have separate treatment for sink nodes and other
> > > nodes, so that other nodes only count the time receiving the record from
> > > last hop? I'm not proposing a solution here, just want to discuss this
> > > alternative to see if it is reasonable.
> > >
> > > 3. As we are going to monitor late arrival records as well, they would
> > > create some really spiky graphs when the out-of-order records are
> > > interleaving with on time records. Should we also supply a smooth version
> > > of the latency metrics, or user should just take care of it by themself?
> > >
> > > 4. Regarding this new metrics, we haven't discussed its relation with our
> > > existing processing latency metrics, could you add some context on
> > > comparison and a simple `when to use which` tutorial for the best?
> > >
> > > Boyang
> > >
> > > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <sophie@confluent.io
> > >
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > I'd like to kick off discussion on KIP-613 which aims to add end-to-end
> > > > latency metrics to Streams. Please take a look:
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > > >
> > > > Cheers,
> > > > Sophie
> > > >
> > >
> >
> 
> 
> -- 
> -- Guozhang
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Guozhang Wang <wa...@gmail.com>.
Thanks Sophie for the KIP, a few quick thoughts:

1) The end-to-end latency includes both the processing latency of the task
and the latency spent sitting in intermediate topics. I have a similar
feeling as Boyang mentioned above that the latency metric of a task A
actually measures the latency of the sub-topology up-to but not including
the processing of A, which is a bit weird.

Maybe the my feeling comes from the name "latency" itself, since today we
already have several "latency" metrics already which are measuring elapsed
system-time for processing a record / etc, while here we are comparing the
system wallclock time with the record timestamp.

Maybe we can consider renaming it as "record-staleness" (note we already
have a "record-lateness" metric), in which case recording at the
system-time before we start processing the record sounds more natural.

2) With that in mind, I'm wondering if the processor-node-level DEBUG
metric is worth to add, given that we already have a task-level processing
latency metric. Basically, a specific node's e2e latency is similar to the
task-level e2e latency + task-level processing latency. Personally I think
having a task-level record-staleness metric is sufficient.



Guozhang



On Wed, May 13, 2020 at 11:46 AM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> 1. I felt that 50% was not a particularly useful gauge for this specific
> metric, as
> it's presumably most useful at putting an *upper *bound on the latency you
> can
> reasonably expect to see. I chose percentiles that would hopefully give a
> good
> sense of what *most* records will experience, and what *close to all*
> records
> will.
>
> However I'm not married to these specific numbers and could be convinced.
> Would be especially interested in hearing from users on this.
>
> 2. I'm inclined to not include the "hop-to-hop latency" in this KIP since
> users
> can always compute it themselves by subtracting the previous node's
> end-to-end latency. I guess we could do it either way since you can always
> compute one from the other, but I think the end-to-end latency feels more
> valuable as it's main motivation is not to debug bottlenecks in the
> topology but
> to give users a sense of how long it takes arecord to be reflected in
> certain parts
> of the topology. For example this might be useful for users who are
> wondering
> roughly when a record that was just produced will be included in their IQ
> results.
> Debugging is just a nice side effect -- but maybe I didn't make that clear
> enough
> in the KIP's motivation.
>
> 3. Good question, I should address this in the KIP. The short answer is
> "yes",
> we will include late records. I added a paragraph to the end of the
> Proposed
> Changes section explaining the reasoning here, please let me know if you
> have
> any concerns.
>
> 4. Assuming you're referring to the existing metric "process-latency", that
> metric
> reflects the time for the literal Node#process method to run whereas this
> metric
> would always be measured relative to the event timestamp.
>
> That said, the naming collision there is pretty confusing so I've renamed
> the
> metrics in this KIP to "end-to-end-latency" which I feel better reflects
> the nature
> of the metric anyway.
>
> Thanks for the feedback!
>
> On Wed, May 13, 2020 at 10:21 AM Boyang Chen <re...@gmail.com>
> wrote:
>
> > Thanks for the KIP Sophie. Getting the E2E latency is important for
> > understanding the bottleneck of the application.
> >
> > A couple of questions and ideas:
> >
> > 1. Could you clarify the rational of picking 75, 99 and max percentiles?
> > Normally I see cases where we use 50, 90 percentile as well in production
> > systems.
> >
> > 2. The current latency being computed is cumulative, I.E if a record goes
> > through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) +
> T(A)
> > and so on, where P() represents the captured latency, and T() represents
> > the time for transiting the records between two nodes, including
> processing
> > time. For monitoring purpose, maybe having T(B->C) and T(A->B) are more
> > natural to view as "hop-to-hop latency", otherwise if there is a spike in
> > T(A->B), both P(B) and P(C) are affected in the same time. In the same
> > spirit, the E2E latency is meaningful only when the record exits from the
> > sink as this marks the whole time this record spent inside the funnel. Do
> > you think we could have separate treatment for sink nodes and other
> > nodes, so that other nodes only count the time receiving the record from
> > last hop? I'm not proposing a solution here, just want to discuss this
> > alternative to see if it is reasonable.
> >
> > 3. As we are going to monitor late arrival records as well, they would
> > create some really spiky graphs when the out-of-order records are
> > interleaving with on time records. Should we also supply a smooth version
> > of the latency metrics, or user should just take care of it by themself?
> >
> > 4. Regarding this new metrics, we haven't discussed its relation with our
> > existing processing latency metrics, could you add some context on
> > comparison and a simple `when to use which` tutorial for the best?
> >
> > Boyang
> >
> > On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <sophie@confluent.io
> >
> > wrote:
> >
> > > Hey all,
> > >
> > > I'd like to kick off discussion on KIP-613 which aims to add end-to-end
> > > latency metrics to Streams. Please take a look:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> > >
> > > Cheers,
> > > Sophie
> > >
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Sophie Blee-Goldman <so...@confluent.io>.
1. I felt that 50% was not a particularly useful gauge for this specific
metric, as
it's presumably most useful at putting an *upper *bound on the latency you
can
reasonably expect to see. I chose percentiles that would hopefully give a
good
sense of what *most* records will experience, and what *close to all* records
will.

However I'm not married to these specific numbers and could be convinced.
Would be especially interested in hearing from users on this.

2. I'm inclined to not include the "hop-to-hop latency" in this KIP since
users
can always compute it themselves by subtracting the previous node's
end-to-end latency. I guess we could do it either way since you can always
compute one from the other, but I think the end-to-end latency feels more
valuable as it's main motivation is not to debug bottlenecks in the
topology but
to give users a sense of how long it takes arecord to be reflected in
certain parts
of the topology. For example this might be useful for users who are
wondering
roughly when a record that was just produced will be included in their IQ
results.
Debugging is just a nice side effect -- but maybe I didn't make that clear
enough
in the KIP's motivation.

3. Good question, I should address this in the KIP. The short answer is
"yes",
we will include late records. I added a paragraph to the end of the Proposed
Changes section explaining the reasoning here, please let me know if you
have
any concerns.

4. Assuming you're referring to the existing metric "process-latency", that
metric
reflects the time for the literal Node#process method to run whereas this
metric
would always be measured relative to the event timestamp.

That said, the naming collision there is pretty confusing so I've renamed
the
metrics in this KIP to "end-to-end-latency" which I feel better reflects
the nature
of the metric anyway.

Thanks for the feedback!

On Wed, May 13, 2020 at 10:21 AM Boyang Chen <re...@gmail.com>
wrote:

> Thanks for the KIP Sophie. Getting the E2E latency is important for
> understanding the bottleneck of the application.
>
> A couple of questions and ideas:
>
> 1. Could you clarify the rational of picking 75, 99 and max percentiles?
> Normally I see cases where we use 50, 90 percentile as well in production
> systems.
>
> 2. The current latency being computed is cumulative, I.E if a record goes
> through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) + T(A)
> and so on, where P() represents the captured latency, and T() represents
> the time for transiting the records between two nodes, including processing
> time. For monitoring purpose, maybe having T(B->C) and T(A->B) are more
> natural to view as "hop-to-hop latency", otherwise if there is a spike in
> T(A->B), both P(B) and P(C) are affected in the same time. In the same
> spirit, the E2E latency is meaningful only when the record exits from the
> sink as this marks the whole time this record spent inside the funnel. Do
> you think we could have separate treatment for sink nodes and other
> nodes, so that other nodes only count the time receiving the record from
> last hop? I'm not proposing a solution here, just want to discuss this
> alternative to see if it is reasonable.
>
> 3. As we are going to monitor late arrival records as well, they would
> create some really spiky graphs when the out-of-order records are
> interleaving with on time records. Should we also supply a smooth version
> of the latency metrics, or user should just take care of it by themself?
>
> 4. Regarding this new metrics, we haven't discussed its relation with our
> existing processing latency metrics, could you add some context on
> comparison and a simple `when to use which` tutorial for the best?
>
> Boyang
>
> On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <so...@confluent.io>
> wrote:
>
> > Hey all,
> >
> > I'd like to kick off discussion on KIP-613 which aims to add end-to-end
> > latency metrics to Streams. Please take a look:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
> >
> > Cheers,
> > Sophie
> >
>

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Posted by Boyang Chen <re...@gmail.com>.
Thanks for the KIP Sophie. Getting the E2E latency is important for
understanding the bottleneck of the application.

A couple of questions and ideas:

1. Could you clarify the rational of picking 75, 99 and max percentiles?
Normally I see cases where we use 50, 90 percentile as well in production
systems.

2. The current latency being computed is cumulative, I.E if a record goes
through A -> B -> C, then P(C) = T(B->C) + P(B) = T(B->C) + T(A->B) + T(A)
and so on, where P() represents the captured latency, and T() represents
the time for transiting the records between two nodes, including processing
time. For monitoring purpose, maybe having T(B->C) and T(A->B) are more
natural to view as "hop-to-hop latency", otherwise if there is a spike in
T(A->B), both P(B) and P(C) are affected in the same time. In the same
spirit, the E2E latency is meaningful only when the record exits from the
sink as this marks the whole time this record spent inside the funnel. Do
you think we could have separate treatment for sink nodes and other
nodes, so that other nodes only count the time receiving the record from
last hop? I'm not proposing a solution here, just want to discuss this
alternative to see if it is reasonable.

3. As we are going to monitor late arrival records as well, they would
create some really spiky graphs when the out-of-order records are
interleaving with on time records. Should we also supply a smooth version
of the latency metrics, or user should just take care of it by themself?

4. Regarding this new metrics, we haven't discussed its relation with our
existing processing latency metrics, could you add some context on
comparison and a simple `when to use which` tutorial for the best?

Boyang

On Tue, May 12, 2020 at 7:28 PM Sophie Blee-Goldman <so...@confluent.io>
wrote:

> Hey all,
>
> I'd like to kick off discussion on KIP-613 which aims to add end-to-end
> latency metrics to Streams. Please take a look:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
>
> Cheers,
> Sophie
>