You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink Jira Bot (Jira)" <ji...@apache.org> on 2022/04/07 10:44:00 UTC
[jira] [Updated] (FLINK-22663) Release YARN resource very slow when cancel the job after some NodeManagers shutdown
[ https://issues.apache.org/jira/browse/FLINK-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flink Jira Bot updated FLINK-22663:
-----------------------------------
Labels: YARN auto-deprioritized-major pull-request-available stale-minor (was: YARN auto-deprioritized-major pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.
> Release YARN resource very slow when cancel the job after some NodeManagers shutdown
> ------------------------------------------------------------------------------------
>
> Key: FLINK-22663
> URL: https://issues.apache.org/jira/browse/FLINK-22663
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.12.2
> Reporter: Jinhong Liu
> Priority: Minor
> Labels: YARN, auto-deprioritized-major, pull-request-available, stale-minor
>
> When I test flink on YARN, there is a case that may cause some problems.
> Hadoop Version: 2.7.3
> Flink Version: 1.12.2
> I deploy a flink job on YARN, when the job is running I stop one NodeManager, after one or two minutes, the job is auto recovered. But in this situation, if I cancel the job, the containers cannot be released immediately, there are still some containers that are running include the app master. About 5 minutes later, these containers exit, and about 10 minutes later the app master exit.
> I check the log of app master, seems it try to stop the containers on the NodeManger which I have already stopped.
> {code:java}
> 2021-05-14 06:15:17,389 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job class tv.freewheel.reporting.fastlane.Fastlane$ (da883ab39a7a82e4d45a3803bc77dd6f) switched from state CANCELLING to CANCELED.
> 2021-05-14 06:15:17,389 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job da883ab39a7a82e4d45a3803bc77dd6f.
> 2021-05-14 06:15:17,390 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down
> 2021-05-14 06:15:17,408 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Job da883ab39a7a82e4d45a3803bc77dd6f reached globally terminal state CANCELED.
> 2021-05-14 06:15:17,409 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Shutting down cluster with state CANCELED, jobCancelled: true, executionMode: DETACHED
> 2021-05-14 06:15:17,409 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting YarnJobClusterEntrypoint down with application status CANCELED. Diagnostics null.
> 2021-05-14 06:15:17,409 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.
> 2021-05-14 06:15:17,420 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job class tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f).
> 2021-05-14 06:15:17,422 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing cache directory /tmp/flink-web-af72a00c-0ddd-4e5e-a62c-8244d6caa552/flink-web-ui
> 2021-05-14 06:15:17,432 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://ip-10-23-19-197.ec2.internal:43811 lost leadership
> 2021-05-14 06:15:17,432 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down complete.
> 2021-05-14 06:15:17,436 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut down cluster because application is in CANCELED, diagnostics null.
> 2021-05-14 06:15:17,436 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Unregister application from the YARN Resource Manager with final status KILLED.
> 2021-05-14 06:15:17,458 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending SlotPool.
> 2021-05-14 06:15:17,458 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection 493862ba148679a4f16f7de5ffaef665: Stopping JobMaster for job class tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f)..
> 2021-05-14 06:15:17,458 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping SlotPool.
> 2021-05-14 06:15:17,482 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl [] - Waiting for application to be successfully unregistered.
> 2021-05-14 06:15:17,566 INFO org.apache.flink.runtime.history.FsJobArchivist [] - Job da883ab39a7a82e4d45a3803bc77dd6f has been archived at hdfs:/realtime/flink-archive/da883ab39a7a82e4d45a3803bc77dd6f.
> 2021-05-14 06:15:17,589 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
> 2021-05-14 06:15:17,590 INFO org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] - Stopping JobDispatcherLeaderProcess.
> 2021-05-14 06:15:17,590 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping dispatcher akka.tcp://flink@ip-10-23-19-197.ec2.internal:40340/user/rpc/dispatcher_1.
> 2021-05-14 06:15:17,590 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping all currently running jobs of dispatcher akka.tcp://flink@ip-10-23-19-197.ec2.internal:40340/user/rpc/dispatcher_1.
> 2021-05-14 06:15:17,591 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator [] - Shutting down back pressure request coordinator.
> 2021-05-14 06:15:17,591 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopped dispatcher akka.tcp://flink@ip-10-23-19-197.ec2.internal:40340/user/rpc/dispatcher_1.
> 2021-05-14 06:15:17,594 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@ip-10-23-19-197.ec2.internal:40340/user/rpc/jobmanager_2 for job da883ab39a7a82e4d45a3803bc77dd6f from the resource manager.
> 2021-05-14 06:15:17,600 INFO org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl [] - Interrupted while waiting for queue
> java.lang.InterruptedException: null
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_161]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_161]
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_161]
> at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287) [flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
> 2021-05-14 06:15:17,648 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:15:17,699 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:15:17,741 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:15:17,796 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:15:17,809 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:15:17,813 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:15:17,817 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:15:17,822 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:15:17,879 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:15:17,889 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
> 2021-05-14 06:15:17,898 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:15:17,903 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:15:17,907 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:15:17,911 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:15:17,960 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
> 2021-05-14 06:15:17,964 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-27-242.ec2.internal:43814] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:17,964 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-27-242.ec2.internal:38826] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,016 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-28-67.ec2.internal:45022] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,016 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-28-67.ec2.internal:40808] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,061 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-139.ec2.internal:33912] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,061 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-139.ec2.internal:44652] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,120 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-71.ec2.internal:42454] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,120 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-71.ec2.internal:41756] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,125 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-27-242.ec2.internal:36652] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,125 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-27-242.ec2.internal:37709] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,126 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-28-67.ec2.internal:40308] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,126 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-28-67.ec2.internal:34524] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,141 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-27-242.ec2.internal:44435] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,141 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-27-242.ec2.internal:37224] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,143 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-71.ec2.internal:38940] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,143 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-71.ec2.internal:33014] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,202 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-19-86.ec2.internal:39939] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,202 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-86.ec2.internal:35165] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,204 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-197.ec2.internal:45913] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,204 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-19-197.ec2.internal:36333] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,220 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-28-67.ec2.internal:35366] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,220 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-28-67.ec2.internal:45411] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,223 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-139.ec2.internal:34759] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,223 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-139.ec2.internal:42621] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,228 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-86.ec2.internal:40782] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,228 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-19-86.ec2.internal:36612] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,251 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-139.ec2.internal:38342] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:15:18,251 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-139.ec2.internal:36176] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:18:18,220 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl [] - Failed to stop Container container_1620970870707_0001_01_000057when stopping NMClientImpl
> 2021-05-14 06:18:18,240 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:18:18,295 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:18:18,299 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
> 2021-05-14 06:18:18,557 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-86.ec2.internal:44399] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:18:18,557 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-19-86.ec2.internal:33246] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:18:18,611 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-71.ec2.internal:39100] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:18:18,611 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-71.ec2.internal:35428] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,684 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-27-242.ec2.internal:41510] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,730 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-197.ec2.internal:39595] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,741 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-24-71.ec2.internal:46788] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,754 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-71.ec2.internal:46748] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,754 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-10-23-24-71.ec2.internal:34218] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:01,761 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-10-23-19-86.ec2.internal:42730] has failed, address is now gated for [50] ms. Reason: [Disassociated]
> 2021-05-14 06:21:18,522 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl [] - Failed to stop Container container_1620970870707_0001_01_000078when stopping NMClientImpl
> 2021-05-14 06:21:18,567 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:21:18,571 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
> 2021-05-14 06:21:18,605 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:21:18,657 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:21:18,698 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:21:18,702 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:21:18,705 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
> 2021-05-14 06:21:18,707 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
> 2021-05-14 06:24:18,934 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl [] - Failed to stop Container container_1620970870707_0001_01_000008when stopping NMClientImpl
> 2021-05-14 06:24:18,986 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:24:19,036 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:24:19,077 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:24:19,080 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:24:19,083 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
> 2021-05-14 06:27:19,303 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl [] - Failed to stop Container container_1620970870707_0001_01_000029when stopping NMClientImpl
> 2021-05-14 06:27:19,349 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:27:19,353 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
> 2021-05-14 06:27:19,402 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:27:19,466 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:27:19,470 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:27:19,504 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:27:19,508 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
> 2021-05-14 06:27:19,510 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
> 2021-05-14 06:27:19,545 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:27:19,548 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
> 2021-05-14 06:27:19,551 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
> 2021-05-14 06:27:19,554 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
> 2021-05-14 06:27:19,557 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
> 2021-05-14 06:27:19,559 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
> 2021-05-14 06:27:50,793 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2021-05-14 06:27:50,794 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:44447
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)