You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by antonio saldivar <an...@gmail.com> on 2018/07/19 20:09:24 UTC
Measure latency from Source to Sink
Hello
I am developing an application using kafka and flink, I need to be able to
measure the latency from the producer and when it comes out to the sink.
I can append the timestamp in Millisecond when I send the trxn from the
producer but at the end how to append the timestamp when it comes out the
sink.
Someone can help me with an example
Thank you
Best Regards
Re: Measure latency from Source to Sink
Posted by antonio saldivar <an...@gmail.com>.
Awesome thank you very much I will check an try with this.. I am pumping
data with a simple Python producer script to send data to just 1 topic
El vie., 20 jul. 2018 a las 11:04, Thakrar, Jayesh (<
jthakrar@conversantmedia.com>) escribió:
> Below are some examples.
>
> They might be old, but the concept is the same - essentially you are
> implementing an interface.
>
> You have to be conscious of not to do heavy or high latency processing to
> be non-impactful to the data flow.
>
>
>
>
> https://www.ashishpaliwal.com/blog/2013/06/flume-cookbook-implementing-custom-interceptors/
>
>
>
>
> https://medium.com/@bkvarda/building-a-custom-flume-interceptor-8c7a55070038
>
>
>
>
>
>
>
> *From: *antonio saldivar <an...@gmail.com>
> *Date: *Friday, July 20, 2018 at 9:57 AM
> *To: *"Thakrar, Jayesh" <jt...@conversantmedia.com>
> *Cc: *"users@kafka.apache.org" <us...@kafka.apache.org>
> *Subject: *Re: Measure latency from Source to Sink
>
>
>
> Hi
>
>
>
> Actually I am running the app I a single node this is for a POC, I was not
> aware of the custom interceptors do you have an example of this?
>
>
>
> Best Regards
>
>
>
> El vie., 20 jul. 2018 a las 9:20, Thakrar, Jayesh (<
> jthakrar@conversantmedia.com>) escribió:
>
> See if you can use a custom interceptors for this.
> The only fuzzy thing is that the clocks would be different so I would be a
> little skeptical of its accuracy.
> I have heard of some companies who have a special topic in which they
> insert test msgs and then read them back - using the same machine for
> producer and consumer, thereby doing away with this dilemma.
> The assumption is that any delay in the cluster would be felt everywhere.
> Another option is to create topics with one replica on all brokers (i.e. #
> of topics = # of brokers) - so you can be sensitive to a single broker
> having an issue.
>
> On 7/19/18, 3:09 PM, "antonio saldivar" <an...@gmail.com> wrote:
>
> Hello
>
> I am developing an application using kafka and flink, I need to be
> able to
> measure the latency from the producer and when it comes out to the
> sink.
>
> I can append the timestamp in Millisecond when I send the trxn from the
> producer but at the end how to append the timestamp when it comes out
> the
> sink.
>
> Someone can help me with an example
>
> Thank you
> Best Regards
>
>
Re: Measure latency from Source to Sink
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Below are some examples.
They might be old, but the concept is the same - essentially you are implementing an interface.
You have to be conscious of not to do heavy or high latency processing to be non-impactful to the data flow.
https://www.ashishpaliwal.com/blog/2013/06/flume-cookbook-implementing-custom-interceptors/
https://medium.com/@bkvarda/building-a-custom-flume-interceptor-8c7a55070038
From: antonio saldivar <an...@gmail.com>
Date: Friday, July 20, 2018 at 9:57 AM
To: "Thakrar, Jayesh" <jt...@conversantmedia.com>
Cc: "users@kafka.apache.org" <us...@kafka.apache.org>
Subject: Re: Measure latency from Source to Sink
Hi
Actually I am running the app I a single node this is for a POC, I was not aware of the custom interceptors do you have an example of this?
Best Regards
El vie., 20 jul. 2018 a las 9:20, Thakrar, Jayesh (<jt...@conversantmedia.com>>) escribió:
See if you can use a custom interceptors for this.
The only fuzzy thing is that the clocks would be different so I would be a little skeptical of its accuracy.
I have heard of some companies who have a special topic in which they insert test msgs and then read them back - using the same machine for producer and consumer, thereby doing away with this dilemma.
The assumption is that any delay in the cluster would be felt everywhere.
Another option is to create topics with one replica on all brokers (i.e. # of topics = # of brokers) - so you can be sensitive to a single broker having an issue.
On 7/19/18, 3:09 PM, "antonio saldivar" <an...@gmail.com>> wrote:
Hello
I am developing an application using kafka and flink, I need to be able to
measure the latency from the producer and when it comes out to the sink.
I can append the timestamp in Millisecond when I send the trxn from the
producer but at the end how to append the timestamp when it comes out the
sink.
Someone can help me with an example
Thank you
Best Regards
Re: Measure latency from Source to Sink
Posted by antonio saldivar <an...@gmail.com>.
Hi
Actually I am running the app I a single node this is for a POC, I was not
aware of the custom interceptors do you have an example of this?
Best Regards
El vie., 20 jul. 2018 a las 9:20, Thakrar, Jayesh (<
jthakrar@conversantmedia.com>) escribió:
> See if you can use a custom interceptors for this.
> The only fuzzy thing is that the clocks would be different so I would be a
> little skeptical of its accuracy.
> I have heard of some companies who have a special topic in which they
> insert test msgs and then read them back - using the same machine for
> producer and consumer, thereby doing away with this dilemma.
> The assumption is that any delay in the cluster would be felt everywhere.
> Another option is to create topics with one replica on all brokers (i.e. #
> of topics = # of brokers) - so you can be sensitive to a single broker
> having an issue.
>
> On 7/19/18, 3:09 PM, "antonio saldivar" <an...@gmail.com> wrote:
>
> Hello
>
> I am developing an application using kafka and flink, I need to be
> able to
> measure the latency from the producer and when it comes out to the
> sink.
>
> I can append the timestamp in Millisecond when I send the trxn from the
> producer but at the end how to append the timestamp when it comes out
> the
> sink.
>
> Someone can help me with an example
>
> Thank you
> Best Regards
>
>
>
Re: Measure latency from Source to Sink
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
See if you can use a custom interceptors for this.
The only fuzzy thing is that the clocks would be different so I would be a little skeptical of its accuracy.
I have heard of some companies who have a special topic in which they insert test msgs and then read them back - using the same machine for producer and consumer, thereby doing away with this dilemma.
The assumption is that any delay in the cluster would be felt everywhere.
Another option is to create topics with one replica on all brokers (i.e. # of topics = # of brokers) - so you can be sensitive to a single broker having an issue.
On 7/19/18, 3:09 PM, "antonio saldivar" <an...@gmail.com> wrote:
Hello
I am developing an application using kafka and flink, I need to be able to
measure the latency from the producer and when it comes out to the sink.
I can append the timestamp in Millisecond when I send the trxn from the
producer but at the end how to append the timestamp when it comes out the
sink.
Someone can help me with an example
Thank you
Best Regards