You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrea Spina <an...@radicalbit.io> on 2019/06/14 07:49:09 UTC
Flink sent/received bytes related metrics: how are they calculated?
Dear Community, I'd like to ask for some details about bytes related
metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes
received *metrics: what I am recording is the following:
I am reading from a Kafka topic *A* records with schema *K* using a source *Sin
*belonging to a pipeline *P1*. The topic *A* is filled with *K* data using
a separated pipeline *p2* and a sink *Sout*.
Basically what I register is that the *bytes sent* metric of *Sin* - available
through the Flink UI -measure twice or more respect of the *bytes received*
of *Sout. *For instance, if my sink Sout records 10GB of *bytes received*
inbound, then my source Sin emits between 20-25GB *bytes sent*.
[image: Screenshot 2019-06-14 at 09.40.08.png]
Is someone able to detail how these two metrics are calculated?
Thank you,
--
*Andrea Spina*
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT
Re: Flink sent/received bytes related metrics: how are they calculated?
Posted by Andrea Spina <an...@radicalbit.io>.
Hi Chesnay, just one downstream: Sin (Source: Enriched Code) outcome is the
right part of the following operator as in the figure; this operator is the
exclusive downstream of Sin.
Thanks,
[image: Screenshot 2019-06-14 at 13.52.52.png]
Il giorno ven 14 giu 2019 alle ore 12:23 Chesnay Schepler <
chesnay@apache.org> ha scritto:
> How does the *P1 *pipeline look like? Are there 2 downstream operators
> reading from *Sin* (in this case the number of bytes would be measured
> twice)?
>
> On 14/06/2019 12:09, Andrea Spina wrote:
>
> Sorry, I totally missed the version: flink-1.6.4, Streaming API
>
> Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler <
> chesnay@apache.org> ha scritto:
>
>> Which version of Flink are you using? There were some issues at some
>> point about double-counting.
>>
>> On 14/06/2019 09:49, Andrea Spina wrote:
>>
>> Dear Community, I'd like to ask for some details about bytes related
>> metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes
>> received *metrics: what I am recording is the following:
>>
>> I am reading from a Kafka topic *A* records with schema *K* using a
>> source *Sin *belonging to a pipeline *P1*. The topic *A* is filled with
>> *K* data using a separated pipeline *p2* and a sink *Sout*.
>>
>> Basically what I register is that the *bytes sent* metric of *Sin* - available
>> through the Flink UI -measure twice or more respect of the *bytes
>> received* of *Sout. *For instance, if my sink Sout records 10GB of *bytes
>> received* inbound, then my source Sin emits between 20-25GB *bytes sent*.
>>
>> [image: Screenshot 2019-06-14 at 09.40.08.png]
>>
>> Is someone able to detail how these two metrics are calculated?
>>
>> Thank you,
>>
>> --
>> *Andrea Spina*
>> Software Engineer @ Radicalbit Srl
>> Via Borsieri 41, 20159, Milano - IT
>>
>>
>>
>
> --
> *Andrea Spina*
> Head of R&D @ Radicalbit Srl
> Via Giovanni Battista Pirelli 11, 20124, Milano - IT
>
>
>
--
*Andrea Spina*
Head of R&D @ Radicalbit Srl
Via Giovanni Battista Pirelli 11, 20124, Milano - IT
Re: Flink sent/received bytes related metrics: how are they
calculated?
Posted by Chesnay Schepler <ch...@apache.org>.
How does the *P1 *pipeline look like? Are there 2 downstream operators
reading from *Sin* (in this case the number of bytes would be measured
twice)?
On 14/06/2019 12:09, Andrea Spina wrote:
> Sorry, I totally missed the version: flink-1.6.4, Streaming API
>
> Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler
> <chesnay@apache.org <ma...@apache.org>> ha scritto:
>
> Which version of Flink are you using? There were some issues at
> some point about double-counting.
>
> On 14/06/2019 09:49, Andrea Spina wrote:
>> Dear Community, I'd like to ask for some details about bytes
>> related metrics in Flink. Precisely, I'm looking at *bytes sent*
>> and *bytes received *metrics: what I am recording is the following:
>>
>> I am reading from a Kafka topic *A* records with schema *K* using
>> a source *Sin *belonging to a pipeline *P1*. The topic *A* is
>> filled with *K* data using a separated pipeline *p2* and a sink
>> *Sout*.
>>
>> Basically what I register is that the /bytes sent/ metric of
>> *Sin* - available through the Flink UI -measure twice or more
>> respect of the /bytes received/ of *Sout. *For instance, if my
>> sink Sout records 10GB of /bytes received/ inbound, then my
>> source Sin emits between 20-25GB /bytes sent/.
>>
>> Screenshot 2019-06-14 at 09.40.08.png
>>
>> Is someone able to detail how these two metrics are calculated?
>>
>> Thank you,
>>
>> --
>> *Andrea Spina*
>> Software Engineer @ Radicalbit Srl
>> Via Borsieri 41, 20159, Milano - IT
>
>
>
>
> --
> *Andrea Spina*
> Head of R&D @ Radicalbit Srl
> Via Giovanni Battista Pirelli 11, 20124, Milano - IT
Re: Flink sent/received bytes related metrics: how are they calculated?
Posted by Andrea Spina <an...@radicalbit.io>.
Sorry, I totally missed the version: flink-1.6.4, Streaming API
Il giorno ven 14 giu 2019 alle ore 11:08 Chesnay Schepler <
chesnay@apache.org> ha scritto:
> Which version of Flink are you using? There were some issues at some point
> about double-counting.
>
> On 14/06/2019 09:49, Andrea Spina wrote:
>
> Dear Community, I'd like to ask for some details about bytes related
> metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes
> received *metrics: what I am recording is the following:
>
> I am reading from a Kafka topic *A* records with schema *K* using a
> source *Sin *belonging to a pipeline *P1*. The topic *A* is filled with
> *K* data using a separated pipeline *p2* and a sink *Sout*.
>
> Basically what I register is that the *bytes sent* metric of *Sin* - available
> through the Flink UI -measure twice or more respect of the *bytes
> received* of *Sout. *For instance, if my sink Sout records 10GB of *bytes
> received* inbound, then my source Sin emits between 20-25GB *bytes sent*.
>
> [image: Screenshot 2019-06-14 at 09.40.08.png]
>
> Is someone able to detail how these two metrics are calculated?
>
> Thank you,
>
> --
> *Andrea Spina*
> Software Engineer @ Radicalbit Srl
> Via Borsieri 41, 20159, Milano - IT
>
>
>
--
*Andrea Spina*
Head of R&D @ Radicalbit Srl
Via Giovanni Battista Pirelli 11, 20124, Milano - IT
Re: Flink sent/received bytes related metrics: how are they
calculated?
Posted by Chesnay Schepler <ch...@apache.org>.
Which version of Flink are you using? There were some issues at some
point about double-counting.
On 14/06/2019 09:49, Andrea Spina wrote:
> Dear Community, I'd like to ask for some details about bytes related
> metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes
> received *metrics: what I am recording is the following:
>
> I am reading from a Kafka topic *A* records with schema *K* using a
> source *Sin *belonging to a pipeline *P1*. The topic *A* is filled
> with *K* data using a separated pipeline *p2* and a sink *Sout*.
>
> Basically what I register is that the /bytes sent/ metric of
> *Sin* - available through the Flink UI -measure twice or more respect
> of the /bytes received/ of *Sout. *For instance, if my sink Sout
> records 10GB of /bytes received/ inbound, then my source Sin emits
> between 20-25GB /bytes sent/.
>
> Screenshot 2019-06-14 at 09.40.08.png
>
> Is someone able to detail how these two metrics are calculated?
>
> Thank you,
>
> --
> *Andrea Spina*
> Software Engineer @ Radicalbit Srl
> Via Borsieri 41, 20159, Milano - IT