[jira] [Commented] (FLINK-8946) TaskManager stop sending metrics after JobManager failover

["Nikolas Davis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501068#comment-16501068 ] 

Nikolas Davis commented on FLINK-8946:
--------------------------------------

I think we are seeing a similar issue. We are using Flink with Mesos. After our Job Manager unexpectedly cycled, we are losing relevant Task Manager JVM metrics. The TMs continue to report _other_ metrics, like operator related ones.

Here are the metric names I see disappearing (sorry, I'm unable to get a good diff of fully qualified metric IDs):


{noformat}
1,8d0
< AvailableMemorySegments
< ClassesLoaded
< ClassesUnloaded
< Committed
< Count
< Load
< Max
< MemoryUsed
10,13d1
< Time
< TotalCapacity
< TotalMemorySegments
< Used{noformat}
 

At first glance these appear mostly to do with memory / JVM, which I suppose are TM-specific.

More info can be found in the [mailing list thread|https://www.mail-archive.com/user@flink.apache.org/msg19289.html].

> TaskManager stop sending metrics after JobManager failover
> ----------------------------------------------------------
>
>                 Key: FLINK-8946
>                 URL: https://issues.apache.org/jira/browse/FLINK-8946
>             Project: Flink
>          Issue Type: Bug
>          Components: Metrics, TaskManager
>    Affects Versions: 1.4.2
>            Reporter: Truong Duc Kien
>            Assignee: vinoyang
>            Priority: Critical
>             Fix For: 1.5.1
>
>
> Running in Yarn-standalone mode, when the Job Manager performs a failover, all TaskManager that are inherited from the previous JobManager will not send metrics to the new JobManager and other registered metric reporters.
>  
> A cursory glance reveal that these line of code might be the cause
> [https://github.com/apache/flink/blob/a3478fdfa0f792104123fefbd9bdf01f5029de51/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala#L1082-L1086]
> Perhap the TaskManager close its metrics group when disassociating JobManager, but not creating a new one on fail-over association ?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)