You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "ynzzxc@163.com" <yn...@163.com> on 2022/07/11 06:45:35 UTC
flink-hudi-hive
Hi，
我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
best,
ynzzxc@163.com
Re: Re: flink-hudi-hive
Posted by "ynzzxc@163.com" <yn...@163.com>.
2022-07-13 06:45:16
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.301-b09 mixed mode):
"Attach Listener" #182 daemon prio=9 os_prio=0 tid=0x00007f1f00641800 nid=0x18ff73 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1" #132 daemon prio=5 os_prio=0 tid=0x000056093ecc7000 nid=0x18ff1a waiting on condition [0x00007f1ef69a2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e090bc98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Jvm Terminator" #24 daemon prio=5 os_prio=0 tid=0x00007f1f09b35000 nid=0x18ff17 waiting on condition [0x00007f1ef1471000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run(JvmShutdownSafeguard.java:82)
at java.lang.Thread.run(Thread.java:748)
"YarnJobClusterEntrypoint shutdown hook" #25 prio=5 os_prio=0 tid=0x00007f1f1c3b7800 nid=0x18ff16 waiting on condition [0x00007f1ef1370000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f8a15520> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3334)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$new$0(ClusterEntrypoint.java:171)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint$$Lambda$68/1404565079.close(Unknown Source)
at org.apache.flink.util.ShutdownHookUtil.lambda$addShutdownHook$0(ShutdownHookUtil.java:39)
at org.apache.flink.util.ShutdownHookUtil$$Lambda$69/652176954.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
"FlinkCompletableFutureDelayScheduler-thread-1" #130 daemon prio=5 os_prio=0 tid=0x00007f1f0033c800 nid=0x18ff13 waiting on condition [0x00007f1ef297a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f7f815a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"LeaseRenewer:dataxc@hdcluster" #127 daemon prio=5 os_prio=0 tid=0x00007f1f02702800 nid=0x18ff0e waiting on condition [0x00007f1ef0362000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:412)
at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
at org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:308)
at java.lang.Thread.run(Thread.java:748)
"org.apache.hadoop.hdfs.PeerCache@6dde409a" #105 daemon prio=5 os_prio=0 tid=0x00007f1f0042d000 nid=0x18fef8 waiting on condition [0x00007f1ef2277000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253)
at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
at java.lang.Thread.run(Thread.java:748)
"connection-pool-192.168.10.124:3306 housekeeper" #98 daemon prio=5 os_prio=0 tid=0x00007f1f0032d800 nid=0x18fef1 waiting on condition [0x00007f1ef2a7b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000feabb1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"mysql-cj-abandoned-connection-cleanup" #97 daemon prio=5 os_prio=0 tid=0x00007f1f007cf800 nid=0x18fef0 in Object.wait() [0x00007f1ef2f7c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000feabb428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000feabb428> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"IPC Client (1833592262) connection to n101/192.168.10.225:9000 from dataxc" #96 daemon prio=5 os_prio=0 tid=0x00007f1f089d2000 nid=0x18feef in Object.wait() [0x00007f1ef327d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000feb70888> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1053)
- locked <0x00000000feb70888> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1097)
"jobmanager-io-thread-1" #95 daemon prio=5 os_prio=0 tid=0x00007f1f08398000 nid=0x18feee waiting on condition [0x00007f1ef397e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e09b0758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-akka.actor.default-dispatcher-22" #94 prio=5 os_prio=0 tid=0x00007f1f1d295000 nid=0x18feed waiting on condition [0x00007f1ef3a7f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0525ed0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.remote.default-remote-dispatcher-21" #93 prio=5 os_prio=0 tid=0x00007f1f083f2800 nid=0x18feec waiting on condition [0x00007f1ef3b80000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.remote.default-remote-dispatcher-20" #92 prio=5 os_prio=0 tid=0x00007f1f1ef6a000 nid=0x18feeb waiting on condition [0x00007f1ef3c81000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-akka.remote.default-remote-dispatcher-17" #88 prio=5 os_prio=0 tid=0x00007f1f1f999800 nid=0x18febe waiting on condition [0x00007f1ef3f84000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e090f4a8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"cluster-io-thread-1" #87 daemon prio=5 os_prio=0 tid=0x00007f1f1f995800 nid=0x18febd waiting on condition [0x00007f1ef4285000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0cf7388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"IPC Client (1833592262) connection to n101/192.168.10.225:8030 from dataxc" #85 daemon prio=5 os_prio=0 tid=0x00007f1f1f939000 nid=0x18feba in Object.wait() [0x00007f1ef4588000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fec5e170> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1053)
- locked <0x00000000fec5e170> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1097)
"flink-akka.actor.default-dispatcher-19" #81 prio=5 os_prio=0 tid=0x00007f1f08a7b000 nid=0x18feb7 waiting on condition [0x00007f1ef4a8b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0525ed0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.actor.default-dispatcher-18" #80 prio=5 os_prio=0 tid=0x00007f1f08a79000 nid=0x18feb6 in Object.wait() [0x00007f1ef4b8b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e008f358> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000e008f358> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:170)
at java.lang.Shutdown.exit(Shutdown.java:216)
- locked <0x00000000e0355148> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:109)
at java.lang.System.exit(System.java:971)
at org.apache.flink.core.security.FlinkSecurityManager.forceProcessExit(FlinkSecurityManager.java:214)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.onFatalError(ClusterEntrypoint.java:469)
at org.apache.flink.runtime.dispatcher.Dispatcher.onFatalError(Dispatcher.java:839)
at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:912)
at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:473)
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$3(Dispatcher.java:430)
at org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$581/1501964576.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$471/156415199.run(Unknown Source)
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$465/1997765037.apply(Unknown Source)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
at akka.actor.ActorCell.invoke(ActorCell.scala:548)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"ForkJoinPool.commonPool-worker-18" #79 daemon prio=5 os_prio=0 tid=0x00007f1f08a73000 nid=0x18feb5 waiting on condition [0x00007f1ef4c8d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0396250> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"Hashed wheel timer #1" #37 prio=5 os_prio=0 tid=0x00007f1f1f8c8800 nid=0x18feb3 sleeping[0x00007f1ef508f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:748)
"flink-akka.remote.default-remote-dispatcher-17" #72 prio=5 os_prio=0 tid=0x000056093f374800 nid=0x18fead waiting on condition [0x00007f1ef5491000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"ForkJoinPool.commonPool-worker-25" #71 daemon prio=5 os_prio=0 tid=0x000056093ef51800 nid=0x18feac waiting on condition [0x00007f1ef5992000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0396250> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-akka.actor.supervisor-dispatcher-16" #70 prio=5 os_prio=0 tid=0x00007f1f1f070800 nid=0x18feab waiting on condition [0x00007f1ef5a93000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0888420> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-metrics-akka.remote.default-remote-dispatcher-15" #69 prio=5 os_prio=0 tid=0x00007f1f1f06d800 nid=0x18feaa waiting on condition [0x00007f1ef5b94000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e090f4a8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"New I/O server boss #12" #68 prio=5 os_prio=0 tid=0x000056093ee45800 nid=0x18fea9 runnable [0x00007f1ef5c95000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e0888c88> (a sun.nio.ch.Util$3)
- locked <0x00000000e0888c98> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e0888c40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #11" #67 prio=5 os_prio=0 tid=0x000056093ecb3800 nid=0x18fea8 runnable [0x00007f1ef5d96000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e089de58> (a sun.nio.ch.Util$3)
- locked <0x00000000e089de68> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e089de10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #10" #66 prio=5 os_prio=0 tid=0x000056093ecb1800 nid=0x18fea7 runnable [0x00007f1ef5e97000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e08896f8> (a sun.nio.ch.Util$3)
- locked <0x00000000e08896e8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e08895d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O boss #9" #65 prio=5 os_prio=0 tid=0x000056093ecaf800 nid=0x18fea6 runnable [0x00007f1ef5f98000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e095aa80> (a sun.nio.ch.Util$3)
- locked <0x00000000e095aa90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e095aa38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #8" #63 prio=5 os_prio=0 tid=0x000056093f1c2000 nid=0x18fea5 runnable [0x00007f1ef6099000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e089e8d8> (a sun.nio.ch.Util$3)
- locked <0x00000000e089e8e8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e089e890> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #7" #62 prio=5 os_prio=0 tid=0x000056093ed7b800 nid=0x18fea4 runnable [0x00007f1ef619a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e092ee28> (a sun.nio.ch.Util$3)
- locked <0x00000000e092ee38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e092ede0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-metrics-akka.remote.default-remote-dispatcher-7" #61 prio=5 os_prio=0 tid=0x00007f1f1f06b800 nid=0x18fea3 waiting on condition [0x00007f1ef629b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e090f4a8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-akka.remote.default-remote-dispatcher-6" #60 prio=5 os_prio=0 tid=0x00007f1f1f069800 nid=0x18fea2 waiting on condition [0x00007f1ef639c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e090f4a8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-5" #59 prio=1 os_prio=0 tid=0x00007f1f1f068000 nid=0x18fea1 waiting on condition [0x00007f1ef649d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e095b180> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-metrics-akka.actor.internal-dispatcher-4" #58 prio=5 os_prio=0 tid=0x00007f1f1f066000 nid=0x18fea0 waiting on condition [0x00007f1ef659e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e095b5d8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-akka.actor.internal-dispatcher-3" #57 prio=5 os_prio=0 tid=0x00007f1f1f064000 nid=0x18fe9f waiting on condition [0x00007f1ef669f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e095b5d8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-akka.actor.internal-dispatcher-2" #56 prio=5 os_prio=0 tid=0x00007f1f1f062000 nid=0x18fe9e waiting on condition [0x00007f1ef67a0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e095b5d8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-metrics-scheduler-1" #55 prio=5 os_prio=0 tid=0x00007f1f1f05a000 nid=0x18fe9d waiting on condition [0x00007f1ef68a1000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.lang.Thread.run(Thread.java:748)
"Curator-Framework-0" #51 daemon prio=5 os_prio=0 tid=0x00007f1f1f02c800 nid=0x18fe9a waiting on condition [0x00007f1ef6da4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e09ad1a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"main-EventThread" #50 daemon prio=5 os_prio=0 tid=0x00007f1f1f027000 nid=0x18fe99 waiting on condition [0x00007f1ef6ea5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0dbfc98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)
"main-SendThread(n102:2181)" #49 daemon prio=5 os_prio=0 tid=0x00007f1f1f000000 nid=0x18fe98 runnable [0x00007f1ef6fa6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e0afcaf0> (a sun.nio.ch.Util$3)
- locked <0x00000000e0afcb00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e0afcaa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
"Curator-ConnectionStateManager-0" #48 daemon prio=5 os_prio=0 tid=0x00007f1f1eff7000 nid=0x18fe97 waiting on condition [0x00007f1ef70a7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e09ad968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)
at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"IPC Parameter Sending Thread #0" #47 daemon prio=5 os_prio=0 tid=0x00007f1f1ef6c800 nid=0x18fe96 waiting on condition [0x00007f1ef71a8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e092fb18> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #44 daemon prio=5 os_prio=0 tid=0x00007f1f1e752000 nid=0x18fe94 in Object.wait() [0x00007f1ef80cc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e0aabcc0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e0aabcc0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4021)
at java.lang.Thread.run(Thread.java:748)
"flink-akka.actor.supervisor-dispatcher-16" #43 prio=5 os_prio=0 tid=0x000056093eca4800 nid=0x18fe93 waiting on condition [0x00007f1ef81cd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e09ab4c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-akka.remote.default-remote-dispatcher-15" #42 prio=5 os_prio=0 tid=0x00007f1f1e69c800 nid=0x18fe92 waiting on condition [0x00007f1ef86ce000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"New I/O server boss #6" #41 prio=5 os_prio=0 tid=0x000056093e862000 nid=0x18fe91 runnable [0x00007f1ef87cf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e04e7328> (a sun.nio.ch.Util$3)
- locked <0x00000000e04e7338> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e04e72e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #5" #40 prio=5 os_prio=0 tid=0x000056093ebee800 nid=0x18fe90 runnable [0x00007f1ef88d0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e04eeea8> (a sun.nio.ch.Util$3)
- locked <0x00000000e04eeeb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e04eee60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #4" #39 prio=5 os_prio=0 tid=0x000056093e569800 nid=0x18fe8f runnable [0x00007f1ef89d1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e04f6e00> (a sun.nio.ch.Util$3)
- locked <0x00000000e04f6e10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e04f6db8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O boss #3" #38 prio=5 os_prio=0 tid=0x000056093e866000 nid=0x18fe8e runnable [0x00007f1ef8ad2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e04fec88> (a sun.nio.ch.Util$3)
- locked <0x00000000e04fec98> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e04fec40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #2" #36 prio=5 os_prio=0 tid=0x000056093ea5f800 nid=0x18fe8d runnable [0x00007f1ef8bd3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e0506c58> (a sun.nio.ch.Util$3)
- locked <0x00000000e0506c68> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e0506c10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"New I/O worker #1" #35 prio=5 os_prio=0 tid=0x000056093ea65800 nid=0x18fe8c runnable [0x00007f1ef8cd4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e050ebf0> (a sun.nio.ch.Util$3)
- locked <0x00000000e050ec00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e050eba8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"flink-akka.remote.default-remote-dispatcher-7" #34 prio=5 os_prio=0 tid=0x00007f1f0050d800 nid=0x18fe8b waiting on condition [0x00007f1ef8fd5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.remote.default-remote-dispatcher-6" #33 prio=5 os_prio=0 tid=0x00007f1f00561000 nid=0x18fe8a waiting on condition [0x00007f1ef90d6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e04df390> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.actor.default-dispatcher-5" #32 prio=5 os_prio=0 tid=0x00007f1f0028d800 nid=0x18fe89 waiting on condition [0x00007f1ef91d7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0525ed0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.actor.internal-dispatcher-4" #31 prio=5 os_prio=0 tid=0x00007f1f1e659800 nid=0x18fe88 waiting on condition [0x00007f1ef92d8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0537410> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.actor.internal-dispatcher-3" #30 prio=5 os_prio=0 tid=0x00007f1f1e654800 nid=0x18fe87 waiting on condition [0x00007f1ef93d9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0537410> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-akka.actor.internal-dispatcher-2" #29 prio=5 os_prio=0 tid=0x00007f1f1e63e000 nid=0x18fe86 waiting on condition [0x00007f1ef94da000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0537410> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1835)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1704)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
"flink-scheduler-1" #28 prio=5 os_prio=0 tid=0x00007f1f1e561800 nid=0x18fe85 sleeping[0x00007f1ef97db000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.lang.Thread.run(Thread.java:748)
"Log4j2-TF-3-Scheduled-2" #27 daemon prio=5 os_prio=0 tid=0x00007f1f1da21000 nid=0x18fe84 waiting on condition [0x00007f1efa491000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e00053e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"process reaper" #21 daemon prio=10 os_prio=0 tid=0x00007f1f1d80c000 nid=0x18fe81 waiting on condition [0x00007f1efa4ca000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0013a00> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Log4j2-TF-2-Scheduled-1" #19 daemon prio=5 os_prio=0 tid=0x00007f1f1d6a8000 nid=0x18fe7f waiting on condition [0x00007f1efd00f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e001b890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f1f1c37f000 nid=0x18fe7c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f1f1c36c000 nid=0x18fe7b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f1f1c369800 nid=0x18fe7a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f1f1c367800 nid=0x18fe79 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f1f1c365800 nid=0x18fe78 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f1f1c363800 nid=0x18fe77 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f1f1c361000 nid=0x18fe76 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f1f1c35f000 nid=0x18fe75 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f1f1c355000 nid=0x18fe74 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f1f1c352800 nid=0x18fe73 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f1f1c350800 nid=0x18fe72 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f1f1c34e800 nid=0x18fe71 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f1f1c34c000 nid=0x18fe70 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f1f1c34a000 nid=0x18fe6f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1f1c317800 nid=0x18fe6e in Object.wait() [0x00007f1f047c6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e002a860> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e002a860> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1f1c30d000 nid=0x18fe6d in Object.wait() [0x00007f1f048c7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e0023d30> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e0023d30> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f1f1c016800 nid=0x18fe59 waiting on condition [0x00007f1f256ed000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e09310a8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3334)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:629)
at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:99)
"VM Thread" os_prio=0 tid=0x00007f1f1c303800 nid=0x18fe6c runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f1f1c02c800 nid=0x18fe5a runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f1f1c02e000 nid=0x18fe5b runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f1f1c030000 nid=0x18fe5c runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f1f1c031800 nid=0x18fe5d runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f1f1c033800 nid=0x18fe5e runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f1f1c035000 nid=0x18fe5f runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f1f1c037000 nid=0x18fe60 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f1f1c038800 nid=0x18fe61 runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f1f1c03a800 nid=0x18fe62 runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f1f1c03c000 nid=0x18fe63 runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f1f1c03e000 nid=0x18fe64 runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f1f1c03f800 nid=0x18fe65 runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f1f1c041800 nid=0x18fe66 runnable
"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f1f1c043000 nid=0x18fe67 runnable
"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f1f1c045000 nid=0x18fe68 runnable
"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f1f1c046800 nid=0x18fe69 runnable
"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f1f1c048800 nid=0x18fe6a runnable
"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f1f1c04a000 nid=0x18fe6b runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f1f1c38c000 nid=0x18fe7d waiting on condition
JNI global references: 2164
best,
ynzzxc@163.com
From: Lijie Wang
Date: 2022-07-13 13:36
To: user-zh
Subject: Re: Re: flink-hudi-hive
Hi，
jstack 下 JM 的栈？ 从目前这个现象上看，像是 JobMaster 初始化的时候卡住了
Best，
Lijie
ynzzxc@163.com <yn...@163.com> 于2022年7月13日周三 09:56写道：
> 是的，192.168.10.227:35961是TM地址；
> 反复初始化是指，在flink web ui的overview界面，Running Job
> List中对应JOb的status一直是INITIALIZING；
> 没有TM日志，我暂时还没弄明白为什么退出，flink web ui的TM界面，全程是没有任何信息的；
> 以下是日志列表，我没找到啥有用信息
> directory.info : Total file length is 7201 bytes.
> jobmanager.err : Total file length is 588 bytes.
> jobmanager.log : Total file length is 82894 bytes.
> jobmanager.out : Total file length is 0 bytes.
> launch_container.sh : Total file length is 21758 bytes.
> prelaunch.err : Total file length is 0 bytes.
> prelaunch.out : Total file length is 100 bytes.
>
>
>
> best,
> ynzzxc@163.com
>
> From: Weihua Hu
> Date: 2022-07-12 23:18
> To: user-zh
> Subject: Re: Re: flink-hudi-hive
> 单从这个日志看不到一直 Failover ，相关任务反复初始化是指哪个任务呢？
> 看到了一些 akka 的链接异常，有可能是对应的 TM 异常退出了，可以再确认下 192.168.10.227:35961 这个是不是
> TaskManager 地址，以及为什么退出
>
> Best,
> Weihua
>
>
> On Tue, Jul 12, 2022 at 9:37 AM ynzzxc@163.com <yn...@163.com> wrote:
>
> > 这是job managers所有日志：
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: execution.shutdown-on-attached-exit, false
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: pipeline.jars,
> > file:/home/dataxc/opt/flink-1.14.4/opt/flink-python_2.11-1.14.4.jar
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: execution.checkpointing.min-pause, 8min
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: restart-strategy, failure-rate
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: jobmanager.memory.jvm-metaspace.size, 128m
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: state.checkpoints.dir, hdfs:///flink/checkpoints
> > 2022-07-12 09:33:02,382 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,383 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> > 2022-07-12 09:33:02,399 INFO
> > org.apache.flink.runtime.rpc.akka.AkkaRpcService [] -
> Starting
> > RPC endpoint for
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
> > akka://flink/user/rpc/resourcemanager_1 .
> > 2022-07-12 09:33:02,405 INFO
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager []
> -
> > Starting the resource manager.
> > 2022-07-12 09:33:02,479 INFO
> > org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] -
> > Failing over to rm2
> > 2022-07-12 09:33:02,509 INFO
> > org.apache.flink.yarn.YarnResourceManagerDriver [] -
> Recovered
> > 0 containers from previous attempts ([]).
> > 2022-07-12 09:33:02,509 INFO
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager []
> -
> > Recovered 0 workers from previous attempt.
> > 2022-07-12 09:33:02,514 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,515 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> > 2022-07-12 09:33:02,528 INFO org.apache.hadoop.conf.Configuration
> > [] - resource-types.xml not found
> > 2022-07-12 09:33:02,528 INFO
> > org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable
> to
> > find 'resource-types.xml'.
> > 2022-07-12 09:33:02,538 INFO
> > org.apache.flink.runtime.externalresource.ExternalResourceUtils [] -
> > Enabled external resources: []
> > 2022-07-12 09:33:02,541 INFO
> > org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper
> > bound of the thread pool size is 500
> > 2022-07-12 09:33:02,584 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,585 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> >
> >
> >
> > best,
> > ynzzxc@163.com
> >
> > From: Weihua Hu
> > Date: 2022-07-11 19:46
> > To: user-zh
> > Subject: Re: flink-hudi-hive
> > Hi,
> > 任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
> > FAILED"
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
> >
> > > Hi，
> > > 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> > > ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> > > 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> > > 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
> > >
> > >
> > >
> > > best,
> > > ynzzxc@163.com
> > >
> >
>
Re: Re: flink-hudi-hive
Posted by Lijie Wang <wa...@gmail.com>.
Hi，
jstack 下 JM 的栈？ 从目前这个现象上看，像是 JobMaster 初始化的时候卡住了
Best，
Lijie
ynzzxc@163.com <yn...@163.com> 于2022年7月13日周三 09:56写道：
> 是的，192.168.10.227:35961是TM地址；
> 反复初始化是指，在flink web ui的overview界面，Running Job
> List中对应JOb的status一直是INITIALIZING；
> 没有TM日志，我暂时还没弄明白为什么退出，flink web ui的TM界面，全程是没有任何信息的；
> 以下是日志列表，我没找到啥有用信息
> directory.info : Total file length is 7201 bytes.
> jobmanager.err : Total file length is 588 bytes.
> jobmanager.log : Total file length is 82894 bytes.
> jobmanager.out : Total file length is 0 bytes.
> launch_container.sh : Total file length is 21758 bytes.
> prelaunch.err : Total file length is 0 bytes.
> prelaunch.out : Total file length is 100 bytes.
>
>
>
> best,
> ynzzxc@163.com
>
> From: Weihua Hu
> Date: 2022-07-12 23:18
> To: user-zh
> Subject: Re: Re: flink-hudi-hive
> 单从这个日志看不到一直 Failover ，相关任务反复初始化是指哪个任务呢？
> 看到了一些 akka 的链接异常，有可能是对应的 TM 异常退出了，可以再确认下 192.168.10.227:35961 这个是不是
> TaskManager 地址，以及为什么退出
>
> Best,
> Weihua
>
>
> On Tue, Jul 12, 2022 at 9:37 AM ynzzxc@163.com <yn...@163.com> wrote:
>
> > 这是job managers所有日志：
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: execution.shutdown-on-attached-exit, false
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: pipeline.jars,
> > file:/home/dataxc/opt/flink-1.14.4/opt/flink-python_2.11-1.14.4.jar
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: execution.checkpointing.min-pause, 8min
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: restart-strategy, failure-rate
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: jobmanager.memory.jvm-metaspace.size, 128m
> > 2022-07-12 09:33:02,280 INFO
> > org.apache.flink.configuration.GlobalConfiguration [] - Loading
> > configuration property: state.checkpoints.dir, hdfs:///flink/checkpoints
> > 2022-07-12 09:33:02,382 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,383 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> > 2022-07-12 09:33:02,399 INFO
> > org.apache.flink.runtime.rpc.akka.AkkaRpcService [] -
> Starting
> > RPC endpoint for
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
> > akka://flink/user/rpc/resourcemanager_1 .
> > 2022-07-12 09:33:02,405 INFO
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager []
> -
> > Starting the resource manager.
> > 2022-07-12 09:33:02,479 INFO
> > org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] -
> > Failing over to rm2
> > 2022-07-12 09:33:02,509 INFO
> > org.apache.flink.yarn.YarnResourceManagerDriver [] -
> Recovered
> > 0 containers from previous attempts ([]).
> > 2022-07-12 09:33:02,509 INFO
> > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager []
> -
> > Recovered 0 workers from previous attempt.
> > 2022-07-12 09:33:02,514 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,515 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> > 2022-07-12 09:33:02,528 INFO org.apache.hadoop.conf.Configuration
> > [] - resource-types.xml not found
> > 2022-07-12 09:33:02,528 INFO
> > org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable
> to
> > find 'resource-types.xml'.
> > 2022-07-12 09:33:02,538 INFO
> > org.apache.flink.runtime.externalresource.ExternalResourceUtils [] -
> > Enabled external resources: []
> > 2022-07-12 09:33:02,541 INFO
> > org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper
> > bound of the thread pool size is 500
> > 2022-07-12 09:33:02,584 WARN akka.remote.transport.netty.NettyTransport
> > [] - Remote connection to [null] failed with
> > java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> > 2022-07-12 09:33:02,585 WARN akka.remote.ReliableDeliverySupervisor
> > [] - Association with remote system
> [akka.tcp://flink@n103:35961]
> > has failed, address is now gated for [50] ms. Reason: [Association failed
> > with [akka.tcp://flink@n103:35961]] Caused by:
> > [java.net.ConnectException: Connection refused: n103/
> 192.168.10.227:35961]
> >
> >
> >
> > best,
> > ynzzxc@163.com
> >
> > From: Weihua Hu
> > Date: 2022-07-11 19:46
> > To: user-zh
> > Subject: Re: flink-hudi-hive
> > Hi,
> > 任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
> > FAILED"
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
> >
> > > Hi，
> > > 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> > > ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> > > 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> > > 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
> > >
> > >
> > >
> > > best,
> > > ynzzxc@163.com
> > >
> >
>
Re: Re: flink-hudi-hive
Posted by "ynzzxc@163.com" <yn...@163.com>.
是的，192.168.10.227:35961是TM地址；
反复初始化是指，在flink web ui的overview界面，Running Job List中对应JOb的status一直是INITIALIZING；
没有TM日志，我暂时还没弄明白为什么退出，flink web ui的TM界面，全程是没有任何信息的；
以下是日志列表，我没找到啥有用信息
directory.info : Total file length is 7201 bytes.
jobmanager.err : Total file length is 588 bytes.
jobmanager.log : Total file length is 82894 bytes.
jobmanager.out : Total file length is 0 bytes.
launch_container.sh : Total file length is 21758 bytes.
prelaunch.err : Total file length is 0 bytes.
prelaunch.out : Total file length is 100 bytes.
best,
ynzzxc@163.com
From: Weihua Hu
Date: 2022-07-12 23:18
To: user-zh
Subject: Re: Re: flink-hudi-hive
单从这个日志看不到一直 Failover ，相关任务反复初始化是指哪个任务呢？
看到了一些 akka 的链接异常，有可能是对应的 TM 异常退出了，可以再确认下 192.168.10.227:35961 这个是不是
TaskManager 地址，以及为什么退出
Best,
Weihua
On Tue, Jul 12, 2022 at 9:37 AM ynzzxc@163.com <yn...@163.com> wrote:
> 这是job managers所有日志：
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.shutdown-on-attached-exit, false
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: pipeline.jars,
> file:/home/dataxc/opt/flink-1.14.4/opt/flink-python_2.11-1.14.4.jar
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.checkpointing.min-pause, 8min
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: restart-strategy, failure-rate
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.memory.jvm-metaspace.size, 128m
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.checkpoints.dir, hdfs:///flink/checkpoints
> 2022-07-12 09:33:02,382 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,383 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
> 2022-07-12 09:33:02,399 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting
> RPC endpoint for
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
> akka://flink/user/rpc/resourcemanager_1 .
> 2022-07-12 09:33:02,405 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Starting the resource manager.
> 2022-07-12 09:33:02,479 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] -
> Failing over to rm2
> 2022-07-12 09:33:02,509 INFO
> org.apache.flink.yarn.YarnResourceManagerDriver [] - Recovered
> 0 containers from previous attempts ([]).
> 2022-07-12 09:33:02,509 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Recovered 0 workers from previous attempt.
> 2022-07-12 09:33:02,514 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,515 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
> 2022-07-12 09:33:02,528 INFO org.apache.hadoop.conf.Configuration
> [] - resource-types.xml not found
> 2022-07-12 09:33:02,528 INFO
> org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to
> find 'resource-types.xml'.
> 2022-07-12 09:33:02,538 INFO
> org.apache.flink.runtime.externalresource.ExternalResourceUtils [] -
> Enabled external resources: []
> 2022-07-12 09:33:02,541 INFO
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper
> bound of the thread pool size is 500
> 2022-07-12 09:33:02,584 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,585 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
>
>
>
> best,
> ynzzxc@163.com
>
> From: Weihua Hu
> Date: 2022-07-11 19:46
> To: user-zh
> Subject: Re: flink-hudi-hive
> Hi,
> 任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
> FAILED"
>
> Best,
> Weihua
>
>
> On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
>
> > Hi，
> > 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> > ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> > 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> > 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
> >
> >
> >
> > best,
> > ynzzxc@163.com
> >
>
Re: Re: flink-hudi-hive
Posted by Weihua Hu <hu...@gmail.com>.
单从这个日志看不到一直 Failover ，相关任务反复初始化是指哪个任务呢？
看到了一些 akka 的链接异常，有可能是对应的 TM 异常退出了，可以再确认下 192.168.10.227:35961 这个是不是
TaskManager 地址，以及为什么退出
Best,
Weihua
On Tue, Jul 12, 2022 at 9:37 AM ynzzxc@163.com <yn...@163.com> wrote:
> 这是job managers所有日志：
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.shutdown-on-attached-exit, false
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: pipeline.jars,
> file:/home/dataxc/opt/flink-1.14.4/opt/flink-python_2.11-1.14.4.jar
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.checkpointing.min-pause, 8min
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: restart-strategy, failure-rate
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.memory.jvm-metaspace.size, 128m
> 2022-07-12 09:33:02,280 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.checkpoints.dir, hdfs:///flink/checkpoints
> 2022-07-12 09:33:02,382 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,383 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
> 2022-07-12 09:33:02,399 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting
> RPC endpoint for
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
> akka://flink/user/rpc/resourcemanager_1 .
> 2022-07-12 09:33:02,405 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Starting the resource manager.
> 2022-07-12 09:33:02,479 INFO
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] -
> Failing over to rm2
> 2022-07-12 09:33:02,509 INFO
> org.apache.flink.yarn.YarnResourceManagerDriver [] - Recovered
> 0 containers from previous attempts ([]).
> 2022-07-12 09:33:02,509 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Recovered 0 workers from previous attempt.
> 2022-07-12 09:33:02,514 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,515 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
> 2022-07-12 09:33:02,528 INFO org.apache.hadoop.conf.Configuration
> [] - resource-types.xml not found
> 2022-07-12 09:33:02,528 INFO
> org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to
> find 'resource-types.xml'.
> 2022-07-12 09:33:02,538 INFO
> org.apache.flink.runtime.externalresource.ExternalResourceUtils [] -
> Enabled external resources: []
> 2022-07-12 09:33:02,541 INFO
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper
> bound of the thread pool size is 500
> 2022-07-12 09:33:02,584 WARN akka.remote.transport.netty.NettyTransport
> [] - Remote connection to [null] failed with
> java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
> 2022-07-12 09:33:02,585 WARN akka.remote.ReliableDeliverySupervisor
> [] - Association with remote system [akka.tcp://flink@n103:35961]
> has failed, address is now gated for [50] ms. Reason: [Association failed
> with [akka.tcp://flink@n103:35961]] Caused by:
> [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
>
>
>
> best,
> ynzzxc@163.com
>
> From: Weihua Hu
> Date: 2022-07-11 19:46
> To: user-zh
> Subject: Re: flink-hudi-hive
> Hi,
> 任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
> FAILED"
>
> Best,
> Weihua
>
>
> On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
>
> > Hi，
> > 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> > ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> > 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> > 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
> >
> >
> >
> > best,
> > ynzzxc@163.com
> >
>
Re: Re: flink-hudi-hive
Posted by "ynzzxc@163.com" <yn...@163.com>.
这是job managers所有日志：
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, file:/home/dataxc/opt/flink-1.14.4/opt/flink-python_2.11-1.14.4.jar
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.checkpointing.min-pause, 8min
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: restart-strategy, failure-rate
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.jvm-metaspace.size, 128m
2022-07-12 09:33:02,280 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, hdfs:///flink/checkpoints
2022-07-12 09:33:02,382 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
2022-07-12 09:33:02,383 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@n103:35961] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@n103:35961]] Caused by: [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
2022-07-12 09:33:02,399 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at akka://flink/user/rpc/resourcemanager_1 .
2022-07-12 09:33:02,405 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Starting the resource manager.
2022-07-12 09:33:02,479 INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to rm2
2022-07-12 09:33:02,509 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Recovered 0 containers from previous attempts ([]).
2022-07-12 09:33:02,509 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Recovered 0 workers from previous attempt.
2022-07-12 09:33:02,514 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
2022-07-12 09:33:02,515 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@n103:35961] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@n103:35961]] Caused by: [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
2022-07-12 09:33:02,528 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found
2022-07-12 09:33:02,528 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'.
2022-07-12 09:33:02,538 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
2022-07-12 09:33:02,541 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper bound of the thread pool size is 500
2022-07-12 09:33:02,584 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: n103/192.168.10.227:35961
2022-07-12 09:33:02,585 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@n103:35961] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@n103:35961]] Caused by: [java.net.ConnectException: Connection refused: n103/192.168.10.227:35961]
best,
ynzzxc@163.com
From: Weihua Hu
Date: 2022-07-11 19:46
To: user-zh
Subject: Re: flink-hudi-hive
Hi,
任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
FAILED"
Best,
Weihua
On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
> Hi，
> 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
>
>
>
> best,
> ynzzxc@163.com
>
Re: flink-hudi-hive
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
任务反复初始化是指一直在 Failover 吗？在 JobManager.log 里可以看到作业 Failover 原因，搜索关键字; "to
FAILED"
Best,
Weihua
On Mon, Jul 11, 2022 at 2:46 PM ynzzxc@163.com <yn...@163.com> wrote:
> Hi，
> 我正在使用flink将数据写入hudi并同步至hive，将任务提交到yarn后，我从flink web
> ui看到：相关任务反复初始化，task managers无任何信息。日志中也无明确错误提示 ；
> 当我删除代码中sync_hive相关配置，并且不改变其他配置，数据能正常写入hudi ；
> 我使用的hudi-0.11.1，flink-1.14.4，hadoop-3.3.1，hive-3.1.3 ；
>
>
>
> best,
> ynzzxc@163.com
>