You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by prateekarora <pr...@gmail.com> on 2016/05/16 21:02:39 UTC
Flink Kafka Streaming - Source [Bytes/records Received] and Sink
[Bytes/records sent] show zero messages
Hi
In my flink kafka streaming application i am fetching data from one topic
and then process and sent to output topic . my application is working fine
but flink dashboard shows Source [Bytes/records Received] and Sink
[Bytes/records sent] is zero.
Duration Name Bytes Records Bytes
Records Status
Received Received
sent sent
3m 15s Source: Custom Source 0 B 0 249 MB
3,540 RUNNING
3m 15s Map -> Sink: Unnamed 203 MB 2,883 0 B 0
RUNNING
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png>
what is the reason for not showing correct number ?
Regards
Prateek
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink Kafka Streaming - Source [Bytes/records Received] and Sink
[Bytes/records sent] show zero messages
Posted by Robert Metzger <rm...@apache.org>.
Hi,
currently, Flink doesn't have code build in for throughput (rate) and
latency measurements. Its a planned feature for the future and some first
steps are done into that direction.
Check out this code for measuring throughput:
https://github.com/rmetzger/flink-state-watermarks/blob/master/src/main/java/com/dataartisans/utils/ThroughputLogger.java
(It will log the rates to the log file)
For the latency, I would attach timestamps to your records and measure the
delay by comparing the timestamps to the current system time (make sure the
clocks on all machines are in sync)
On Mon, May 16, 2016 at 11:43 PM, prateekarora <pr...@gmail.com>
wrote:
> thanks for the information .
> I want to collect some performance metrics like throughput , latency etc
> .thats why looking for these data to collect some information .
>
> how can i collect some performance data in flink ?
>
> Regards
> Prateek
>
>
> On Mon, May 16, 2016 at 2:32 PM, Fabian Hueske-2 [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=6945&i=0>> wrote:
>
>> Hi Prateek,
>>
>> the missing numbers are an artifact from how the stats are collected.
>> ATM, Flink does only collect these metrics for data which is sent over
>> connections *between* Flink operators.
>> Since sources and sinks connect to external systems (and not Flink
>> operators), the dash board does not show receiving byte for sources and
>> sent bytes for sinks.
>>
>> Best, Fabian
>>
>> 2016-05-16 23:02 GMT+02:00 prateekarora <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=6944&i=0>>:
>>
>>> Hi
>>>
>>> In my flink kafka streaming application i am fetching data from one topic
>>> and then process and sent to output topic . my application is working
>>> fine
>>> but flink dashboard shows Source [Bytes/records Received] and Sink
>>> [Bytes/records sent] is zero.
>>>
>>>
>>> Duration Name Bytes Records
>>> Bytes
>>> Records Status
>>> Received Received
>>> sent sent
>>> 3m 15s Source: Custom Source 0 B 0
>>> 249 MB
>>> 3,540 RUNNING
>>>
>>>
>>> 3m 15s Map -> Sink: Unnamed 203 MB 2,883 0 B
>>> 0
>>> RUNNING
>>>
>>>
>>>
>>> <
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png
>>> >
>>>
>>> what is the reason for not showing correct number ?
>>>
>>> Regards
>>> Prateek
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6944.html
>> To unsubscribe from Flink Kafka Streaming - Source [Bytes/records
>> Received] and Sink [Bytes/records sent] show zero messages, click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Flink Kafka Streaming - Source
> [Bytes/records Received] and Sink [Bytes/records sent] show zero messages
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6945.html>
>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
Re: Flink Kafka Streaming - Source [Bytes/records Received] and
Sink [Bytes/records sent] show zero messages
Posted by prateekarora <pr...@gmail.com>.
thanks for the information .
I want to collect some performance metrics like throughput , latency etc
.thats why looking for these data to collect some information .
how can i collect some performance data in flink ?
Regards
Prateek
On Mon, May 16, 2016 at 2:32 PM, Fabian Hueske-2 [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi Prateek,
>
> the missing numbers are an artifact from how the stats are collected.
> ATM, Flink does only collect these metrics for data which is sent over
> connections *between* Flink operators.
> Since sources and sinks connect to external systems (and not Flink
> operators), the dash board does not show receiving byte for sources and
> sent bytes for sinks.
>
> Best, Fabian
>
> 2016-05-16 23:02 GMT+02:00 prateekarora <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=6944&i=0>>:
>
>> Hi
>>
>> In my flink kafka streaming application i am fetching data from one topic
>> and then process and sent to output topic . my application is working fine
>> but flink dashboard shows Source [Bytes/records Received] and Sink
>> [Bytes/records sent] is zero.
>>
>>
>> Duration Name Bytes Records
>> Bytes
>> Records Status
>> Received Received
>> sent sent
>> 3m 15s Source: Custom Source 0 B 0
>> 249 MB
>> 3,540 RUNNING
>>
>>
>> 3m 15s Map -> Sink: Unnamed 203 MB 2,883 0 B
>> 0
>> RUNNING
>>
>>
>>
>> <
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png
>> >
>>
>> what is the reason for not showing correct number ?
>>
>> Regards
>> Prateek
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6944.html
> To unsubscribe from Flink Kafka Streaming - Source [Bytes/records
> Received] and Sink [Bytes/records sent] show zero messages, click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=6941&code=cHJhdGVlay5hcm9yYTJrNkBnbWFpbC5jb218Njk0MXwtOTMxMzI0ODYx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6945.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink Kafka Streaming - Source [Bytes/records Received] and Sink
[Bytes/records sent] show zero messages
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Prateek,
the missing numbers are an artifact from how the stats are collected.
ATM, Flink does only collect these metrics for data which is sent over
connections *between* Flink operators.
Since sources and sinks connect to external systems (and not Flink
operators), the dash board does not show receiving byte for sources and
sent bytes for sinks.
Best, Fabian
2016-05-16 23:02 GMT+02:00 prateekarora <pr...@gmail.com>:
> Hi
>
> In my flink kafka streaming application i am fetching data from one topic
> and then process and sent to output topic . my application is working fine
> but flink dashboard shows Source [Bytes/records Received] and Sink
> [Bytes/records sent] is zero.
>
>
> Duration Name Bytes Records
> Bytes
> Records Status
> Received Received
> sent sent
> 3m 15s Source: Custom Source 0 B 0 249
> MB
> 3,540 RUNNING
>
>
> 3m 15s Map -> Sink: Unnamed 203 MB 2,883 0 B
> 0
> RUNNING
>
>
>
> <
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png
> >
>
> what is the reason for not showing correct number ?
>
> Regards
> Prateek
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>