End to end latency of storm topology

[preethini v <pr...@gmail.com>]

Hi,

Is there a way to measure the latency of storm system apart from the
parameter "complete latency" from Storm UI?

I cannot enable the acking, hence complete latency is always 0. I am
looking for other ways to measure end to end latency.

Can we attach a timestamp when a tuple is emitted and check its end time in
execute() method of my last bolt? I am trying to do this, but I do not see
a way to attach timestamp to the tuple and send it further downstream.

Any hints to measure the latency would be very helpful. Thanks.

Best,
Preethini

Re: End to end latency of storm topology

[Federico D'Ambrosio <fe...@gmail.com>]

Hi, if you're using the Trident APIs something like this should be enough:

stream.each( new Fields(), new Timestamper(), new Fields("timestamp")),
where TimeStamper is derived from BaseFunction where you do something
like collector.emit(new
Values(new Date()) in the executor(tuple, collector) method.

2017-07-20 22:19 GMT+02:00 preethini v <pr...@gmail.com>:

> Hi,
>
> Is there a way to measure the latency of storm system apart from the
> parameter "complete latency" from Storm UI?
>
> I cannot enable the acking, hence complete latency is always 0. I am
> looking for other ways to measure end to end latency.
>
> Can we attach a timestamp when a tuple is emitted and check its end time
> in execute() method of my last bolt? I am trying to do this, but I do not
> see a way to attach timestamp to the tuple and send it further downstream.
>
> Any hints to measure the latency would be very helpful. Thanks.
>
> Best,
> Preethini
>



-- 
Federico D'Ambrosio

Re: End to end latency of storm topology

[Jungtaek Lim <ka...@gmail.com>]

Where Acker excels is that Acker tracks 'tuple tree' so it takes care about
branching, filtering, and so on. Whatever operation you add to the
topology, Acker knows when tuple tree is considered as 'complete'. If we
add branch and filter to the topology without acker, it makes really hard
to determine when the tuple is processed completely.
I'm not sure other frameworks can calculate this correctly, given that they
don't have component for tracking tuple. If they can it would be great to
know how they're doing.

Adding timestamp will work with simple topologies (no branch, no filter),
but has another requirement that clock should be strictly synced between
nodes. I saw some docs claiming that time skew between nodes could be
hundreds of milliseconds even though nodes are in same LAN and both are
syncing from same ntp server. Time skew might mess up end to end latency
and you need to tolerate or ensure that is really minimum. (We're using
milliseconds as unit for complete latency, so even a small time skew can
contribute.)

We have long discussion thread regarding improving complete latency applied
to 1.0.3/1.1.0/2.0.0. You might get some ideas while following up this
thread.
http://mail-archives.apache.org/mod_mbox/storm-dev/201604.mbox/%3cCAF5108gn=rSkUNdfs7-sgY_pD-_prgJ2hF2T5e5ZPpP-KnD-hg@mail.gmail.com%3e

Hope this helps.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 7월 21일 (금) 오전 5:49, Bobby Evans <ev...@yahoo-inc.com>님이 작성:

> There is no built in way right now.  You would have to copy the time stamp
> in each stage in your topology.  If you want to contribute something like
> that to storm it would be interesting.
>
>
>
> - Bobby
>
>
>
> On Thursday, July 20, 2017, 3:19:38 PM CDT, preethini v <
> preethini.v@gmail.com> wrote:
>
>
> Hi,
>
> Is there a way to measure the latency of storm system apart from the
> parameter "complete latency" from Storm UI?
>
> I cannot enable the acking, hence complete latency is always 0. I am
> looking for other ways to measure end to end latency.
>
> Can we attach a timestamp when a tuple is emitted and check its end time
> in execute() method of my last bolt? I am trying to do this, but I do not
> see a way to attach timestamp to the tuple and send it further downstream.
>
> Any hints to measure the latency would be very helpful. Thanks.
>
> Best,
> Preethini
>

Re: End to end latency of storm topology

[Bobby Evans <ev...@yahoo-inc.com>]

There is no built in way right now.  You would have to copy the time stamp in each stage in your topology.  If you want to contribute something like that to storm it would be interesting.


- Bobby


On Thursday, July 20, 2017, 3:19:38 PM CDT, preethini v <pr...@gmail.com> wrote:

Hi,
Is there a way to measure the latency of storm system apart from the parameter "complete latency" from Storm UI?
I cannot enable the acking, hence complete latency is always 0. I am looking for other ways to measure end to end latency. 
Can we attach a timestamp when a tuple is emitted and check its end time in execute() method of my last bolt? I am trying to do this, but I do not see a way to attach timestamp to the tuple and send it further downstream.
Any hints to measure the latency would be very helpful. Thanks.
Best,Preethini