You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Ken (Jira)" <ji...@apache.org> on 2019/10/29 19:41:00 UTC
[jira] [Comment Edited] (FLINK-11127) Make metrics query service
establish connection to JobManager
[ https://issues.apache.org/jira/browse/FLINK-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962337#comment-16962337 ]
Ken edited comment on FLINK-11127 at 10/29/19 7:40 PM:
-------------------------------------------------------
[~hwanju], I upgraded from 1.7 to 1.8. Previously I use the workaround fix. With 1.8, I noticed that
{code:java}
Association with remote system [akka.tcp://flink-metrics@<JOB_MANAGER_HOSTNAME>:<PORT> has failed
{code}
has surface again. This does not happen when I start the JM and TMs at the beginning. I was able to force this error by simply "cancel" the job and let it auto-restart. I tried both the workaround as well as removing
{code:java}
taskmanager.host{code}
and use
{code:java}
taskmanager.network.bind-policy: ip{code}
Whenever I "cancel" the job through the UI, the warning shows up in JM and TMs, and the statistics and metrics in the UI stopped working.
On a side note, I noticed that the TM is trying to connect to the JM through a different IP (looks to me like an external IP that's not accessible). Also I am working in a kubernetes environment.
was (Author: tsubasa2oo2):
[~hwanju], I upgraded from 1.7 to 1.8. Previously I use the workaround fix. With 1.8, I noticed that
{code:java}
Association with remote system [akka.tcp://flink-metrics@<JOB_MANAGER_HOSTNAME>:<PORT> has failed
{code}
has surface again. This does not happen when I start the JM and TMs at the beginning. I was able to force this error by simply "cancel" the job and let it auto-restart. I tried both the workaround as well as removing
{code:java}
taskmanager.host{code}
and use
{code:java}
taskmanager.network.bind-policy: ip{code}
Whenever I "cancel" the job through the UI, the warning shows up in JM and TMs, and the statistics and metrics in the UI stopped working.
> Make metrics query service establish connection to JobManager
> -------------------------------------------------------------
>
> Key: FLINK-11127
> URL: https://issues.apache.org/jira/browse/FLINK-11127
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / Coordination, Runtime / Metrics
> Affects Versions: 1.7.0
> Reporter: Ufuk Celebi
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of FLINK-10247, the internal metrics query service has been separated into its own actor system. Before this change, the JobManager (JM) queried TaskManager (TM) metrics via the TM actor. Now, the JM needs to establish a separate connection to the TM metrics query service actor.
> In the context of Kubernetes, this is problematic as the JM will typically *not* be able to resolve the TMs by name, resulting in warnings as follows:
> {code}
> 2018-12-11 08:32:33,962 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183]] Caused by: [flink-task-manager-64b868487c-x9l4b: Name does not resolve]
> {code}
> In order to expose the TMs by name in Kubernetes, users require a service *for each* TM instance which is not practical.
> This currently results in the web UI not being to display some basic metrics about number of sent records. You can reproduce this by following the READMEs in {{flink-container/kubernetes}}.
> This worked before, because the JM is typically exposed via a service with a known name and the TMs establish the connection to it which the metrics query service piggybacked on.
> A potential solution to this might be to let the query service connect to the JM similar to how the TMs register.
> I tagged this ticket as an improvement, but in the context of Kubernetes I would consider this to be a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)