You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Abdul Qadeer (JIRA)" <ji...@apache.org> on 2019/05/07 23:04:00 UTC
[jira] [Created] (FLINK-12437) Taskmanager doesn't initiate
registration after jobmanager marks it terminated
Abdul Qadeer created FLINK-12437:
------------------------------------
Summary: Taskmanager doesn't initiate registration after jobmanager marks it terminated
Key: FLINK-12437
URL: https://issues.apache.org/jira/browse/FLINK-12437
Project: Flink
Issue Type: Bug
Reporter: Abdul Qadeer
This issue is observed in Standalone cluster deployment mode with Zookeeper HA enabled in Flink 1.4.0. A few taskmanagers restarted due to Out of Metaspace.
The offending taskmanager `pipelineruntime-taskmgr-6789dd578b-dcp4r` first successfully registers with jobmanager, and the remote watcher marks it terminated soon after as seen in logs. There were other taskmanagers that were terminated around same time but they had been quarantined by jobmanager with message similar to:
{noformat}
Association to [akka.tcp://flink@10.60.5.121:8070] having UID [864976677] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.
{noformat}
They came back up and successfully registered with jobmanager. This didn't happen for the offending taskmanager:
At JobManager:
{noformat}
{"timeMillis":1557073368155,"thread":"flink-akka.actor.default-dispatcher-49","level":"INFO","loggerName":"org.apache.flink.runtime.instance.InstanceManager","message":"Registered TaskManager at pipelineruntime-taskmgr-6789dd578b-dcp4r (akka.tcp://flink@10.60.5.85:8070/user/taskmanager) as ae61ac607f0ab35ab5066f7dc221e654. Current number of registered hosts is 8. Current number of alive task slots is 51.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":125,"threadPriority":5}
...
...
{"timeMillis":1557073391386,"thread":"flink-akka.actor.default-dispatcher-82","level":"INFO","loggerName":"org.apache.flink.runtime.instance.InstanceManager","message":"Unregistered task manager /10.60.5.85. Number of registered task managers 7. Number of available slots 45.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":159,"threadPriority":5}
...
...
{"timeMillis":1557073391483,"thread":"flink-akka.actor.default-dispatcher-82","level":"INFO","loggerName":"org.apache.flink.runtime.instance.InstanceManager","message":"Unregistered task manager /10.60.5.85. Number of registered task managers 6. Number of available slots 39.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":159,"threadPriority":5}
...
...
{"timeMillis":1557073370389,"thread":"flink-akka.actor.default-dispatcher-35","level":"INFO","loggerName":"akka.actor.LocalActorRef","message":"Message [akka.remote.ReliableDeliverySupervisor$Ungate$] from Actor[akka://flink/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fflink%4010.60.5.85%3A8070-3#1863607260] to Actor[akka://flink/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fflink%4010.60.5.85%3A8070-3#1863607260] was not delivered. [22] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":98,"threadPriority":5}
{noformat}
At TaskManager:
{noformat}
{"timeMillis":1557073366068,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Starting TaskManager","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073366073,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Starting TaskManager actor system at 10.60.5.85:8070.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073366077,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Trying to start actor system at 10.60.5.85:8070","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073366510,"thread":"flink-akka.actor.default-dispatcher-4","level":"INFO","loggerName":"akka.event.slf4j.Slf4jLogger","message":"Slf4jLogger started","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":51,"threadPriority":5}
{"timeMillis":1557073366694,"thread":"flink-akka.actor.default-dispatcher-4","level":"INFO","loggerName":"akka.remote.Remoting","message":"Starting remoting","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":51,"threadPriority":5}
{"timeMillis":1557073367049,"thread":"flink-akka.actor.default-dispatcher-4","level":"INFO","loggerName":"akka.remote.Remoting","message":"Remoting started; listening on addresses :[akka.tcp://flink@10.60.5.85:8070]","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":51,"threadPriority":5}
{"timeMillis":1557073367051,"thread":"flink-akka.actor.default-dispatcher-4","level":"INFO","loggerName":"akka.remote.Remoting","message":"Remoting now listens on addresses: [akka.tcp://flink@10.60.5.85:8070]","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":51,"threadPriority":5}
{"timeMillis":1557073367089,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Actor system started at akka.tcp://flink@10.60.5.85:8070","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367138,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.metrics.MetricRegistryImpl","message":"Configuring FlinkMetricsReporter with {class=com.cisco.ndp.pipeline.processor.flink.metrics.FlinkMetricsReporter}.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367139,"thread":"pool-2-thread-1","level":"INFO","loggerName":"com.cisco.ndp.pipeline.processor.flink.metrics.FlinkMetricsReporter","message":"Metrics Reporter Open","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367139,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.metrics.MetricRegistryImpl","message":"Reporting metrics for reporter ndp of type com.cisco.ndp.pipeline.processor.flink.metrics.FlinkMetricsReporter.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367142,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Starting TaskManager actor","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367176,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.network.netty.NettyConfig","message":"NettyConfig [server address: /10.60.5.85, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 3 (manual), number of client threads: 3 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367187,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration","message":"Messages have a max timeout of 100000 ms","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367198,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskexecutor.TaskManagerServices","message":"Temporary file directory '/tmp': total 373 GB, usable 295 GB (79.09% usable)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367608,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.network.buffer.NetworkBufferPool","message":"Allocated 639 MB for network buffer pool (number of memory segments: 20467, bytes per segment: 32768).","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367710,"thread":"pool-2-thread-1","level":"WARN","loggerName":"org.apache.flink.runtime.query.QueryableStateUtils","message":"Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. Please put the corresponding jar from the opt to the lib folder.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367711,"thread":"pool-2-thread-1","level":"WARN","loggerName":"org.apache.flink.runtime.query.QueryableStateUtils","message":"Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. Please put the corresponding jar from the opt to the lib folder.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367712,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.network.NetworkEnvironment","message":"Starting the network environment and its components.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367753,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.network.netty.NettyClient","message":"Successful initialization (took 34 ms).","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367805,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.network.netty.NettyServer","message":"Successful initialization (took 51 ms). Listening on SocketAddress /10.60.5.85:38873.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367808,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.taskexecutor.TaskManagerServices","message":"Limiting managed memory to 0.7 of the currently free heap space (4005 MB), memory will be allocated lazily.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367819,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.io.disk.iomanager.IOManager","message":"I/O manager uses directory /tmp/flink-io-5f657721-13dd-40aa-9c00-2a15d5666280 for spill files.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367826,"thread":"pool-2-thread-1","level":"INFO","loggerName":"org.apache.flink.runtime.filecache.FileCache","message":"User file cache uses directory /tmp/flink-dist-cache-30b1f2fd-9457-435b-a601-ae0b4e37dc6d","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":40,"threadPriority":5}
{"timeMillis":1557073367862,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.filecache.FileCache","message":"User file cache uses directory /tmp/flink-dist-cache-3dfb3cd5-b261-4df3-a662-a1cd91047c72","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367888,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Starting TaskManager actor at akka://flink/user/taskmanager#1157564383.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367889,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"TaskManager data connection information: pipelineruntime-taskmgr-6789dd578b-dcp4r-57b5f60d8144eb16425ec5bd9666768f @ pipelineruntime-taskmgr-6789dd578b-dcp4r (dataPort=38873)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367890,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"TaskManager has 6 task slot(s).","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367892,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Memory usage stats: [HEAP: 842/6554/6554 MB, NON HEAP: 62/64/1776 MB (used/committed/max)]","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367892,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService","message":"Starting ZooKeeperLeaderRetrievalService.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073367965,"thread":"pool-2-thread-1-EventThread","level":"INFO","loggerName":"org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService","message":"Leader node has changed with Leader=akka.tcp://flink@10.60.5.53:6123/user/jobmanager, session ID=270a3383-8f1e-4f2d-b1d6-f7af727e9ea0.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":46,"threadPriority":5}
{"timeMillis":1557073367966,"thread":"pool-2-thread-1-EventThread","level":"INFO","loggerName":"org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService","message":"New leader information: Leader=akka.tcp://flink@10.60.5.53:6123/user/jobmanager, session ID=270a3383-8f1e-4f2d-b1d6-f7af727e9ea0.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":46,"threadPriority":5}
{"timeMillis":1557073367975,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Trying to register at JobManager akka.tcp://flink@10.60.5.53:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073368168,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Successful registration at JobManager (akka.tcp://flink@10.60.5.53:6123/user/jobmanager), starting network stack and library cache.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073368177,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.taskmanager.TaskManager","message":"Determined BLOB server address to be /10.60.5.53:43987. Starting BLOB cache.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073368184,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.blob.PermanentBlobCache","message":"Created BLOB cache storage directory /tmp/blobStore-ffdc49ba-e86f-4240-93ad-7566c43e9b0d","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073368189,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"org.apache.flink.runtime.blob.TransientBlobCache","message":"Created BLOB cache storage directory /tmp/blobStore-764277b6-6e46-4c8f-b7ee-80f746edefab","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391398,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$R4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391399,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$S4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391399,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$T4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391400,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$U4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391400,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$V4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391401,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$W4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391401,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$X4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391474,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$Y4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391475,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$Z4] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
{"timeMillis":1557073391477,"thread":"flink-akka.actor.default-dispatcher-3","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$04] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":50,"threadPriority":5}
...
...
...
{"timeMillis":1557073691534,"thread":"flink-akka.actor.default-dispatcher-2","level":"INFO","loggerName":"akka.actor.EmptyLocalActorRef","message":"Message [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] from Actor[akka.tcp://flink@10.60.5.53:6123/temp/$sab] to Actor[akka://flink/user/taskmanager#-1883282689] was not delivered. [316] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":49,"threadPriority":5}
{noformat}
TCP dump at taskmanager:
{noformat}
19:55:58.214944 IP 10.60.5.85.45008 > 10.60.5.53.6123: tcp 715
0x0000: 4500 02ff 2809 4000 4006 f0ee 0a3c 0555 E...(.@.@....<.U
0x0010: 0a3c 0535 afd0 17eb a107 10ac 0270 79da .<.5.........py.
0x0020: 8018 ce96 21f3 0000 0101 080a f2c0 c93f ....!..........?
0x0030: b74c ec05 0000 02c7 0ac4 0512 c105 0a3d .L.............=
0x0040: 0a3b 616b 6b61 2e74 6370 3a2f 2f66 6c69 .;akka.tcp://fli
0x0050: 6e6b 4031 302e 3630 2e35 2e35 333a 3631 nk@10.60.5.53:61
0x0060: 3233 2f75 7365 722f 6a6f 626d 616e 6167 23/user/jobmanag
0x0070: 6572 2331 3231 3433 3237 3831 3312 bf04 er#1214327813...
0x0080: 0aba 04ac ed00 0573 7200 3f6f 7267 2e61 .......sr.?org.a
0x0090: 7061 6368 652e 666c 696e 6b2e 7275 6e74 pache.flink.runt
0x00a0: 696d 652e 6d65 7373 6167 6573 2e54 6173 ime.messages.Tas
0x00b0: 6b4d 616e 6167 6572 4d65 7373 6167 6573 kManagerMessages
0x00c0: 2448 6561 7274 6265 6174 1fb7 fffd 259b $Heartbeat....%.
0x00d0: c539 0200 024c 000c 6163 6375 6d75 6c61 .9...L..accumula
0x00e0: 746f 7273 7400 164c 7363 616c 612f 636f torst..Lscala/co
0x00f0: 6c6c 6563 7469 6f6e 2f53 6571 3b4c 000a llection/Seq;L..
0x0100: 696e 7374 616e 6365 4944 7400 2e4c 6f72 instanceIDt..Lor
0x0110: 672f 6170 6163 6865 2f66 6c69 6e6b 2f72 g/apache/flink/r
0x0120: 756e 7469 6d65 2f69 6e73 7461 6e63 652f untime/instance/
0x0130: 496e 7374 616e 6365 4944 3b78 7073 7200 InstanceID;xpsr.
0x0140: 2473 6361 6c61 2e63 6f6c 6c65 6374 696f $scala.collectio
0x0150: 6e2e 6d75 7461 626c 652e 4172 7261 7942 n.mutable.ArrayB
0x0160: 7566 6665 7215 38b0 5383 828e 7302 0003 uffer.8.S...s...
0x0170: 4900 0b69 6e69 7469 616c 5369 7a65 4900 I..initialSizeI.
0x0180: 0573 697a 6530 5b00 0561 7272 6179 7400 .size0[..arrayt.
0x0190: 135b 4c6a 6176 612f 6c61 6e67 2f4f 626a .[Ljava/lang/Obj
0x01a0: 6563 743b 7870 0000 0010 0000 0000 7572 ect;xp........ur
0x01b0: 0013 5b4c 6a61 7661 2e6c 616e 672e 4f62 ..[Ljava.lang.Ob
0x01c0: 6a65 6374 3b90 ce58 9f10 7329 6c02 0000 ject;..X..s)l...
0x01d0: 7870 0000 0010 7070 7070 7070 7070 7070 xp....pppppppppp
0x01e0: 7070 7070 7070 7372 002c 6f72 672e 6170 ppppppsr.,org.ap
0x01f0: 6163 6865 2e66 6c69 6e6b 2e72 756e 7469 ache.flink.runti
0x0200: 6d65 2e69 6e73 7461 6e63 652e 496e 7374 me.instance.Inst
0x0210: 616e 6365 4944 0000 0000 0000 0001 0200 anceID..........
0x0220: 0078 7200 206f 7267 2e61 7061 6368 652e .xr..org.apache.
0x0230: 666c 696e 6b2e 7574 696c 2e41 6273 7472 flink.util.Abstr
0x0240: 6163 7449 4400 0000 0000 0000 0102 0003 actID...........
0x0250: 4a00 096c 6f77 6572 5061 7274 4a00 0975 J..lowerPartJ..u
0x0260: 7070 6572 5061 7274 4c00 0874 6f53 7472 pperPartL..toStr
0x0270: 696e 6774 0012 4c6a 6176 612f 6c61 6e67 ingt..Ljava/lang
0x0280: 2f53 7472 696e 673b 7870 ae61 ac60 7f0a /String;xp.a.`..
0x0290: b35a b506 6f7d c221 e654 7400 2061 6536 .Z..o}.!.Tt..ae6
0x02a0: 3161 6336 3037 6630 6162 3335 6162 3530 1ac607f0ab35ab50
0x02b0: 3636 6637 6463 3232 3165 3635 3410 0122 66f7dc221e654.."
0x02c0: 3e0a 3c61 6b6b 612e 7463 703a 2f2f 666c >.<akka.tcp://fl
0x02d0: 696e 6b40 3130 2e36 302e 352e 3835 3a38 ink@10.60.5.85:8
0x02e0: 3037 302f 7573 6572 2f74 6173 6b6d 616e 070/user/taskman
0x02f0: 6167 6572 2331 3135 3735 3634 3338 33 ager#1157564383
19:55:58.214996 IP 10.60.5.53.6123 > 10.60.5.85.45008: tcp 0
0x0000: 4500 0034 c1fe 4000 3f06 5ac4 0a3c 0535 E..4..@.?.Z..<.5
0x0010: 0a3c 0555 17eb afd0 0270 79da a107 1377 .<.U.....py....w
0x0020: 8010 ce93 1f28 0000 0101 080a b74c ff8d .....(.......L..
0x0030: f2c0 c93f ...?
{noformat}
After this, the taskmanager never registers again at the jobmanager.
This run had the following akka configuration:
akka.watch.heartbeat.pause: 60 s
akka.ask.timeout: 100 s
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)