metrics for Flink sinks

[Martin Eden <ma...@gmail.com>]

Hi all,

Just 3 quick questions both related to Flink metrics, especially around
sinks:

1. In the Flink UI Sources always have 0 input records / bytes and Sinks
always have 0 output records / bytes? Why is it like that?

2. What is the best practice for instrumenting off the shelf Flink sinks?

Currently the only metrics available are num records/bytes in and out at
the operator and task scope. For the task scope there are extra buffer
metrics. However the output metrics are always zero (see question 1). How
can one know the actual number of successful writes done by an off the
shelf Flink sink? Or the latency of the write operation?

3. Is it possible to configure Flink to get global job metrics for all
subtasks of an operator? Or are there any best practices around that?

Thanks,
M

Re: metrics for Flink sinks

[Elias Levy <fe...@gmail.com>]

Not an exact match, but I am guessing it is related to FLINK-7286
<https://issues.apache.org/jira/projects/FLINK/issues/FLINK-7286>, which I
reported.  Feel free to modify that issue to cover the root cause.

On Wed, Aug 30, 2017 at 8:32 AM, Martin Eden <ma...@gmail.com>
wrote:

> Thanks Chesnay,
>
> Just for completeness, are there any relevant tickets for the discussion
> that one can follow, upvote, contribute to?
>
> M
>
> On Tue, Aug 29, 2017 at 8:57 PM, Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> Hello,
>>
>> 1. Because no one found time to fix it. In contrast to the remaining
>> byte/record metrics, input metrics for sources / output metrics for sinks
>> have to be implemented for every single implementation with their
>> respective semantics. In contrast, the output metrics are gathered in the
>> intersection between operators, independent of the actual operator
>> implementation. Furthermore, this requires system metrics (i.e. metrics
>> that Flink itself creates) to be exposed (and be mutable!) to user-defined
>> functions, which is something i *generally *wanted to avoid, but it
>> appears to be a big enough pain point to make an exception here.
>>
>> 2. Due to the above it is currently not possible without modifications of
>> the code to know how many reads/writes were made.
>>
>> 3. Do you mean aggregated metrics? The web UI allows the aggregation of
>> record/byte metrics on the task level. Beyond that we defer aggregation to
>> actual time-series databases that specialize in these things.
>>
>>
>> On 28.08.2017 19:08, Martin Eden wrote:
>>
>> Hi all,
>>
>> Just 3 quick questions both related to Flink metrics, especially around
>> sinks:
>>
>> 1. In the Flink UI Sources always have 0 input records / bytes and Sinks
>> always have 0 output records / bytes? Why is it like that?
>>
>> 2. What is the best practice for instrumenting off the shelf Flink sinks?
>>
>> Currently the only metrics available are num records/bytes in and out at
>> the operator and task scope. For the task scope there are extra buffer
>> metrics. However the output metrics are always zero (see question 1). How
>> can one know the actual number of successful writes done by an off the
>> shelf Flink sink? Or the latency of the write operation?
>>
>> 3. Is it possible to configure Flink to get global job metrics for all
>> subtasks of an operator? Or are there any best practices around that?
>>
>> Thanks,
>> M
>>
>>
>>
>

Re: metrics for Flink sinks

[Martin Eden <ma...@gmail.com>]

Thanks Chesnay,

Just for completeness, are there any relevant tickets for the discussion
that one can follow, upvote, contribute to?

M

On Tue, Aug 29, 2017 at 8:57 PM, Chesnay Schepler <ch...@apache.org>
wrote:

> Hello,
>
> 1. Because no one found time to fix it. In contrast to the remaining
> byte/record metrics, input metrics for sources / output metrics for sinks
> have to be implemented for every single implementation with their
> respective semantics. In contrast, the output metrics are gathered in the
> intersection between operators, independent of the actual operator
> implementation. Furthermore, this requires system metrics (i.e. metrics
> that Flink itself creates) to be exposed (and be mutable!) to user-defined
> functions, which is something i *generally *wanted to avoid, but it
> appears to be a big enough pain point to make an exception here.
>
> 2. Due to the above it is currently not possible without modifications of
> the code to know how many reads/writes were made.
>
> 3. Do you mean aggregated metrics? The web UI allows the aggregation of
> record/byte metrics on the task level. Beyond that we defer aggregation to
> actual time-series databases that specialize in these things.
>
>
> On 28.08.2017 19:08, Martin Eden wrote:
>
> Hi all,
>
> Just 3 quick questions both related to Flink metrics, especially around
> sinks:
>
> 1. In the Flink UI Sources always have 0 input records / bytes and Sinks
> always have 0 output records / bytes? Why is it like that?
>
> 2. What is the best practice for instrumenting off the shelf Flink sinks?
>
> Currently the only metrics available are num records/bytes in and out at
> the operator and task scope. For the task scope there are extra buffer
> metrics. However the output metrics are always zero (see question 1). How
> can one know the actual number of successful writes done by an off the
> shelf Flink sink? Or the latency of the write operation?
>
> 3. Is it possible to configure Flink to get global job metrics for all
> subtasks of an operator? Or are there any best practices around that?
>
> Thanks,
> M
>
>
>

Re: metrics for Flink sinks

[Chesnay Schepler <ch...@apache.org>]

Hello,

1. Because no one found time to fix it. In contrast to the remaining 
byte/record metrics, input metrics for sources / output metrics for 
sinks have to be implemented for every single implementation with their 
respective semantics. In contrast, the output metrics are gathered in 
the intersection between operators, independent of the actual operator 
implementation. Furthermore, this requires system metrics (i.e. metrics 
that Flink itself creates) to be exposed (and be mutable!) to 
user-defined functions, which is something i /generally /wanted to 
avoid, but it appears to be a big enough pain point to make an exception 
here.

2. Due to the above it is currently not possible without modifications 
of the code to know how many reads/writes were made.

3. Do you mean aggregated metrics? The web UI allows the aggregation of 
record/byte metrics on the task level. Beyond that we defer aggregation 
to actual time-series databases that specialize in these things.

On 28.08.2017 19:08, Martin Eden wrote:
> Hi all,
>
> Just 3 quick questions both related to Flink metrics, especially 
> around sinks:
>
> 1. In the Flink UI Sources always have 0 input records / bytes and 
> Sinks always have 0 output records / bytes? Why is it like that?
>
> 2. What is the best practice for instrumenting off the shelf Flink sinks?
>
> Currently the only metrics available are num records/bytes in and out 
> at the operator and task scope. For the task scope there are extra 
> buffer metrics. However the output metrics are always zero (see 
> question 1). How can one know the actual number of successful writes 
> done by an off the shelf Flink sink? Or the latency of the write 
> operation?
>
> 3. Is it possible to configure Flink to get global job metrics for all 
> subtasks of an operator? Or are there any best practices around that?
>
> Thanks,
> M