[jira] [Comment Edited] (FLINK-14164) Add a metric to show failover count regarding fine grained recovery

["Zhu Zhu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956735#comment-16956735 ] 

Zhu Zhu edited comment on FLINK-14164 at 10/30/19 2:33 AM:
-----------------------------------------------------------

Make this issue a subtask of FLINK-10429 to also add such a metric for NG scheduler.

I plan to change the {{SchedulerBase}} to register a meter 'numberOfRestarts' to exhibit all restarts.
The meter is a {{MeterView}} and the underlying counter is determined by each scheduler implementation：
1. for legacy scheduler, it's the {{ExecutionGraph#numberOfRestartsCounter}} we added in FLINK-14206
2. for ng scheduler, it's a new counter added in {{ExecutionFailureHandler}} that counts all the task and global failures notified to it (based on FLINK-14232 global failure handling).

[~trohrmann] [~gjy] Do you think this would work? If it's Ok, could you assign this ticket to me?


was (Author: zhuzh):
Make this issue a subtask of FLINK-10429 to also add such a metric for NG scheduler.

I plan to change the {{SchedulerBase}} to register a meter 'numberOfRestarts' to exhibits all restarts.
The meter is a {{MeterView}} and the underlying counter is determined by each scheduler implementation：
1. for legacy scheduler, it's the {{ExecutionGraph#numberOfRestartsCounter}} we added in FLINK-10429
2. for ng scheduler, it's a new counter added in {{ExecutionFailureHandler}} that counts all the task and global failures notified to it (based on FLINK-14232 global failure handling).

[~trohrmann] [~gjy] Do you think this would work? If it's Ok, could you assign this ticket to me?

> Add a metric to show failover count regarding fine grained recovery
> -------------------------------------------------------------------
>
>                 Key: FLINK-14164
>                 URL: https://issues.apache.org/jira/browse/FLINK-14164
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination, Runtime / Metrics
>    Affects Versions: 1.10.0
>            Reporter: Zhu Zhu
>            Priority: Major
>             Fix For: 1.10.0
>
>
> Previously Flink uses restart all strategy to recover jobs from failures. And the metric "fullRestart" is used to show the count of failovers.
> However, with fine grained recovery introduced in 1.9.0, the "fullRestart" metric only reveals how many times the entire graph has been restarted, not including the number of fine grained failure recoveries.
> As many users want to build their job alerting based on failovers, I'd propose to add such a new metric {{numberOfFailures}}/{{numberOfRestarts}} which also respects fine grained recoveries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)