You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (Jira)" <ji...@apache.org> on 2020/09/14 21:16:00 UTC
[jira] [Commented] (FLINK-18128)
CoordinatedSourceITCase.testMultipleSources gets stuck
[ https://issues.apache.org/jira/browse/FLINK-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195735#comment-17195735 ]
Stephan Ewen commented on FLINK-18128:
--------------------------------------
The problem is the following:
- When the fetching thread gets the fetch that signals "end of split" it enqueues that into the handover queue
- Then there may be a context switch to the reader thread that checks this fetch and finishes its own split handling and checks whether the split fetcher is idle. It it not at that point.
- The result is that the reader eventually goes to MORE_AVAILABLE_LATER
- The split fetcher thread continues and marks the split fetcher as idle. However, it will never be checked again, because the reader thread is never notified again of any availability so it never checks.
*Solution part 1:*
The split fetcher needs to notify the availability future when it turns idle. I think there is no way to guarantee that the fetcher gets checked for being idle otherwise, save for introducing locking around the enqueuing and the idleness check. That is more expensive, the notification is rare and cheap.
*Solution part 2:*
The current model how the availability future is implemented makes this still dead-lock dangerous: The reader thread (mailbox thread) creates the availability future when obtaining it. It may be that the notification comes before the thread obtains the future and the notification is lost.
Consider this scenario:
- Fetcher Thread gets "end of split" it enqueues that into the handover queue
- Reader Thread obtains this fetch and removes the split locally. Split Reader is not idle, yet, Reader Thread concludes MORE_AVAILABLE_LATER
- Fetcher Thread notifies the FutureNotifier - no future has been requested so far, the notification does nothing.
- Reader Thread gets the future from the FutureNotifier, which will now never be complete.
We could fix this by adding a dummy element into the {{FutureCompletingBlockingQueue}}, but I think that FLINK-19223 is a better solution when looking at the bigger picture.
> CoordinatedSourceITCase.testMultipleSources gets stuck
> ------------------------------------------------------
>
> Key: FLINK-18128
> URL: https://issues.apache.org/jira/browse/FLINK-18128
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
> Affects Versions: 1.11.0
> Reporter: Robert Metzger
> Assignee: Stephan Ewen
> Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2705&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=6b04ca5f-0b52-511d-19c9-52bf0d9fbdfa
> {code}
> 2020-06-04T11:19:39.6335702Z [INFO]
> 2020-06-04T11:19:39.6337440Z [INFO] -------------------------------------------------------
> 2020-06-04T11:19:39.6338176Z [INFO] T E S T S
> 2020-06-04T11:19:39.6339305Z [INFO] -------------------------------------------------------
> 2020-06-04T11:19:40.1906157Z [INFO] Running org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase
> 2020-06-04T11:34:51.0599860Z ==============================================================================
> 2020-06-04T11:34:51.0603015Z Maven produced no output for 900 seconds.
> 2020-06-04T11:34:51.0604174Z ==============================================================================
> 2020-06-04T11:34:51.0613908Z ==============================================================================
> 2020-06-04T11:34:51.0615097Z The following Java processes are running (JPS)
> 2020-06-04T11:34:51.0616043Z ==============================================================================
> 2020-06-04T11:34:51.0762007Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.2775341Z 29635 surefirebooter5307550588991461882.jar
> 2020-06-04T11:34:51.2931264Z 2100 Launcher
> 2020-06-04T11:34:51.3012583Z 32203 Jps
> 2020-06-04T11:34:51.3258038Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.5443730Z ==============================================================================
> 2020-06-04T11:34:51.5445134Z Printing stack trace of Java process 29635
> 2020-06-04T11:34:51.5445984Z ==============================================================================
> 2020-06-04T11:34:51.5528602Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.9617670Z 2020-06-04 11:34:51
> 2020-06-04T11:34:51.9619131Z Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):
> 2020-06-04T11:34:51.9619732Z
> 2020-06-04T11:34:51.9620618Z "Attach Listener" #299 daemon prio=9 os_prio=0 tid=0x00007f4d60001000 nid=0x7e59 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:51.9621720Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:51.9622190Z
> 2020-06-04T11:34:51.9623631Z "flink-akka.actor.default-dispatcher-185" #297 prio=5 os_prio=0 tid=0x00007f4ca0003000 nid=0x7db4 waiting on condition [0x00007f4d10136000]
> 2020-06-04T11:34:51.9624972Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9625716Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9627072Z - parking to wait for <0x0000000080c557f0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9628593Z at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 2020-06-04T11:34:51.9629649Z at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9630825Z at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9631559Z
> 2020-06-04T11:34:51.9633020Z "flink-akka.actor.default-dispatcher-186" #298 prio=5 os_prio=0 tid=0x00007f4d08006800 nid=0x7db3 waiting on condition [0x00007f4d12974000]
> 2020-06-04T11:34:51.9634074Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9634965Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9636384Z - parking to wait for <0x0000000080c557f0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9637683Z at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 2020-06-04T11:34:51.9638795Z at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9639845Z at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9640475Z
> 2020-06-04T11:34:51.9642008Z "flink-akka.actor.default-dispatcher-182" #293 prio=5 os_prio=0 tid=0x00007f4ce4007000 nid=0x7d7f waiting on condition [0x00007f4d1063b000]
> 2020-06-04T11:34:51.9643266Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:51.9644081Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9645784Z - parking to wait for <0x0000000080c557f0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9647361Z at akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 2020-06-04T11:34:51.9648516Z at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 2020-06-04T11:34:51.9649692Z at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9651016Z at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9651739Z
> 2020-06-04T11:34:51.9653311Z "jobmanager-future-thread-16" #236 daemon prio=5 os_prio=0 tid=0x00007f4c94002000 nid=0x7a99 waiting on condition [0x00007f4d12873000]
> 2020-06-04T11:34:51.9654578Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9655376Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9656968Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9658360Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9659770Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9661421Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9663047Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9664605Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9665917Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9667296Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9668392Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9668914Z
> 2020-06-04T11:34:51.9670561Z "Flink-DispatcherRestEndpoint-thread-4" #119 daemon prio=5 os_prio=0 tid=0x00007f4cc0004800 nid=0x749b waiting on condition [0x00007f4d1328f000]
> 2020-06-04T11:34:51.9671805Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9672620Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9674216Z - parking to wait for <0x0000000087803238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9675411Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9676984Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9678652Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9680193Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9681672Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9682904Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9684149Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9685116Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9685563Z
> 2020-06-04T11:34:51.9687028Z "Flink-DispatcherRestEndpoint-thread-3" #118 daemon prio=5 os_prio=0 tid=0x00007f4cc0003800 nid=0x7490 waiting on condition [0x00007f4c825e4000]
> 2020-06-04T11:34:51.9688144Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9688792Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9690067Z - parking to wait for <0x0000000087803238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9691152Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9692335Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9693745Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9695311Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9696509Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9697640Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9698724Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9699618Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9700045Z
> 2020-06-04T11:34:51.9701396Z "Flink-DispatcherRestEndpoint-thread-2" #115 daemon prio=5 os_prio=0 tid=0x00007f4cc0002800 nid=0x7478 waiting on condition [0x00007f4d12f78000]
> 2020-06-04T11:34:51.9702417Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9703084Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9704565Z - parking to wait for <0x0000000087803238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9705653Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9706804Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9708230Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9709574Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9710779Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9711890Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9713162Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9714226Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9714854Z
> 2020-06-04T11:34:51.9715760Z "SourceFetcher" #112 prio=5 os_prio=0 tid=0x00007f4c10006800 nid=0x746b waiting on condition [0x00007f4c812d3000]
> 2020-06-04T11:34:51.9716852Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9717754Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9719392Z - parking to wait for <0x000000008777dc80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9726297Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9727728Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9729254Z at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> 2020-06-04T11:34:51.9730516Z at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> 2020-06-04T11:34:51.9731853Z at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:113)
> 2020-06-04T11:34:51.9733084Z at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:94)
> 2020-06-04T11:34:51.9734253Z at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:42)
> 2020-06-04T11:34:51.9735413Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2020-06-04T11:34:51.9736573Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-06-04T11:34:51.9737863Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-06-04T11:34:51.9739214Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9740349Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9740883Z
> 2020-06-04T11:34:51.9743025Z "SourceCoordinator-Source: TestingSource2" #110 prio=5 os_prio=0 tid=0x00007f4cf406a000 nid=0x7469 waiting on condition [0x00007f4c814d5000]
> 2020-06-04T11:34:51.9744816Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9745675Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9747526Z - parking to wait for <0x0000000087a7aff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9749566Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9751072Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9752591Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:51.9753926Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9755414Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9756776Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9757999Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9758523Z
> 2020-06-04T11:34:51.9759948Z "SourceCoordinator-Source: TestingSource1" #109 prio=5 os_prio=0 tid=0x00007f4cf4068800 nid=0x7468 waiting on condition [0x00007f4c815d6000]
> 2020-06-04T11:34:51.9760998Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9761673Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9762927Z - parking to wait for <0x0000000087200178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9764012Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9765377Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9766639Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:51.9767882Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9769195Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9770567Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9771666Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9772235Z
> 2020-06-04T11:34:51.9773191Z "Sink: Unnamed (2/4)" #105 prio=5 os_prio=0 tid=0x00007f4cf4060800 nid=0x7464 waiting on condition [0x00007f4c81bda000]
> 2020-06-04T11:34:51.9774776Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9775591Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9777459Z - parking to wait for <0x0000000087a7b278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9778794Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9780275Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9782114Z at org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:146)
> 2020-06-04T11:34:51.9783693Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:279)
> 2020-06-04T11:34:51.9785319Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:190)
> 2020-06-04T11:34:51.9786679Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
> 2020-06-04T11:34:51.9788033Z at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:557)
> 2020-06-04T11:34:51.9789165Z at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:529)
> 2020-06-04T11:34:51.9790155Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:720)
> 2020-06-04T11:34:51.9791088Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:545)
> 2020-06-04T11:34:51.9791879Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9792357Z
> 2020-06-04T11:34:51.9793428Z "OutputFlusher for Source: TestingSource2" #98 daemon prio=5 os_prio=0 tid=0x00007f4c10001800 nid=0x745d waiting on condition [0x00007f4c822e1000]
> 2020-06-04T11:34:51.9794635Z java.lang.Thread.State: TIMED_WAITING (sleeping)
> 2020-06-04T11:34:51.9795319Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:51.9796321Z at org.apache.flink.runtime.io.network.api.writer.RecordWriter$OutputFlusher.run(RecordWriter.java:334)
> 2020-06-04T11:34:51.9797036Z
> 2020-06-04T11:34:51.9797921Z "Source: TestingSource2 (2/4)" #94 prio=5 os_prio=0 tid=0x00007f4cf4055800 nid=0x7459 waiting on condition [0x00007f4c826e5000]
> 2020-06-04T11:34:51.9798908Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9799555Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9800954Z - parking to wait for <0x0000000087500178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9802006Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9803187Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9804559Z at org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:146)
> 2020-06-04T11:34:51.9805858Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:279)
> 2020-06-04T11:34:51.9807228Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:190)
> 2020-06-04T11:34:51.9808529Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
> 2020-06-04T11:34:51.9809766Z at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:557)
> 2020-06-04T11:34:51.9810856Z at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:529)
> 2020-06-04T11:34:51.9811862Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:720)
> 2020-06-04T11:34:51.9812794Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:545)
> 2020-06-04T11:34:51.9813580Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9814025Z
> 2020-06-04T11:34:51.9814876Z "CloseableReaperThread" #89 daemon prio=5 os_prio=0 tid=0x00007f4c28004800 nid=0x7454 in Object.wait() [0x00007f4c82bea000]
> 2020-06-04T11:34:51.9815918Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-06-04T11:34:51.9816734Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:51.9817885Z - waiting on <0x00000000878034a0> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:51.9818759Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-06-04T11:34:51.9819928Z - locked <0x00000000878034a0> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:51.9820808Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-06-04T11:34:51.9821937Z at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:204)
> 2020-06-04T11:34:51.9822706Z
> 2020-06-04T11:34:51.9823940Z "Flink-MetricRegistry-thread-1" #87 daemon prio=5 os_prio=0 tid=0x00007f4cf403a000 nid=0x7452 waiting on condition [0x00007f4c82dec000]
> 2020-06-04T11:34:51.9825063Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:51.9825768Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9827061Z - parking to wait for <0x0000000080c46498> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9828195Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:51.9829405Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:51.9830790Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:51.9832105Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9833408Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9834537Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9835667Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9836568Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9837003Z
> 2020-06-04T11:34:51.9838290Z "jobmanager-future-thread-15" #86 daemon prio=5 os_prio=0 tid=0x00007f4d000e4800 nid=0x7451 waiting on condition [0x00007f4c82eed000]
> 2020-06-04T11:34:51.9839293Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9839970Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9841223Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9842294Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9843451Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9844893Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9846266Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9847511Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9848582Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9849641Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9850531Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9850957Z
> 2020-06-04T11:34:51.9852196Z "jobmanager-future-thread-14" #85 daemon prio=5 os_prio=0 tid=0x00007f4d000e2800 nid=0x7450 waiting on condition [0x00007f4c82fee000]
> 2020-06-04T11:34:51.9853177Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9853845Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9855174Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9856287Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9857618Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9858956Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9860293Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9861465Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9862544Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9863626Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9864587Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9865040Z
> 2020-06-04T11:34:51.9866307Z "jobmanager-future-thread-13" #84 daemon prio=5 os_prio=0 tid=0x00007f4d000e1000 nid=0x744f waiting on condition [0x00007f4c830ef000]
> 2020-06-04T11:34:51.9867374Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9868019Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9869286Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9870338Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9871506Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9872859Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9874275Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9875582Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9876638Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9877780Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9878683Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9879109Z
> 2020-06-04T11:34:51.9880327Z "jobmanager-future-thread-12" #83 daemon prio=5 os_prio=0 tid=0x00007f4d000df800 nid=0x744e waiting on condition [0x00007f4c831f0000]
> 2020-06-04T11:34:51.9881409Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9882126Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9883427Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9884608Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9885796Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9887213Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9888559Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9889735Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9890801Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9891859Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9892754Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9893187Z
> 2020-06-04T11:34:51.9894424Z "jobmanager-future-thread-11" #82 daemon prio=5 os_prio=0 tid=0x00007f4d000dd800 nid=0x744d waiting on condition [0x00007f4c832f1000]
> 2020-06-04T11:34:51.9895512Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9896193Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9897529Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9898711Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9899878Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9901271Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9902744Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9904043Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9905324Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9906518Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9907468Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9907922Z
> 2020-06-04T11:34:51.9909177Z "jobmanager-future-thread-10" #81 daemon prio=5 os_prio=0 tid=0x00007f4d000db800 nid=0x744c waiting on condition [0x00007f4c833f2000]
> 2020-06-04T11:34:51.9910178Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9910825Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9912091Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9913164Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9914314Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9915996Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9917386Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9918590Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9919657Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9920718Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9921606Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9922033Z
> 2020-06-04T11:34:51.9923282Z "jobmanager-future-thread-9" #80 daemon prio=5 os_prio=0 tid=0x00007f4d000da000 nid=0x744b waiting on condition [0x00007f4c834f3000]
> 2020-06-04T11:34:51.9924257Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9925107Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9926445Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9927613Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9928764Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9930131Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9931469Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9932647Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9933721Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9934914Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9935852Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9936281Z
> 2020-06-04T11:34:51.9937591Z "jobmanager-future-thread-8" #79 daemon prio=5 os_prio=0 tid=0x00007f4d000d8000 nid=0x744a waiting on condition [0x00007f4c835f4000]
> 2020-06-04T11:34:51.9938593Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9939359Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9940645Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9941698Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9942880Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9944220Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9945780Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9946985Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9948097Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9949185Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9950060Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9950506Z
> 2020-06-04T11:34:51.9951732Z "jobmanager-future-thread-7" #78 daemon prio=5 os_prio=0 tid=0x00007f4d000d6800 nid=0x7449 waiting on condition [0x00007f4c836f5000]
> 2020-06-04T11:34:51.9952738Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9953383Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9954784Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9956012Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9957224Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9958589Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9959909Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9961100Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9962169Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9963229Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9964122Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9964684Z
> 2020-06-04T11:34:51.9965965Z "jobmanager-future-thread-6" #77 daemon prio=5 os_prio=0 tid=0x00007f4d000d5000 nid=0x7448 waiting on condition [0x00007f4c837f6000]
> 2020-06-04T11:34:51.9966947Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9967675Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9968927Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9970017Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9971191Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9972532Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9973873Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9975208Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9976321Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9977472Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9978347Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9978776Z
> 2020-06-04T11:34:51.9980010Z "jobmanager-future-thread-5" #76 daemon prio=5 os_prio=0 tid=0x00007f4d000d3800 nid=0x7447 waiting on condition [0x00007f4c838f7000]
> 2020-06-04T11:34:51.9981144Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:51.9981812Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9983091Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9984161Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:51.9985490Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:51.9986902Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:51.9988288Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9989486Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9990539Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9991623Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9992519Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9992945Z
> 2020-06-04T11:34:51.9994158Z "jobmanager-future-thread-4" #75 daemon prio=5 os_prio=0 tid=0x00007f4d000d1000 nid=0x7446 waiting on condition [0x00007f4c839f8000]
> 2020-06-04T11:34:51.9995240Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9996082Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9997404Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9998474Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9999624Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0000989Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0002324Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0003501Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0004656Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0005750Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0006656Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0007083Z
> 2020-06-04T11:34:52.0008361Z "mini-cluster-io-thread-14" #74 daemon prio=5 os_prio=0 tid=0x00007f4cf402b000 nid=0x7445 waiting on condition [0x00007f4c83af9000]
> 2020-06-04T11:34:52.0009334Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0010002Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0011256Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0012329Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0013501Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0014777Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0015971Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0017254Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0018345Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0019239Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0019668Z
> 2020-06-04T11:34:52.0020806Z "pool-2-thread-1" #73 prio=5 os_prio=0 tid=0x00007f4cf4028800 nid=0x7444 waiting on condition [0x00007f4c83bfa000]
> 2020-06-04T11:34:52.0021869Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0022542Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0023806Z - parking to wait for <0x000000008797ae48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0024972Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0026160Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0027597Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:52.0028931Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0030107Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0031193Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0032255Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0033149Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0033574Z
> 2020-06-04T11:34:52.0034882Z "jobmanager-future-thread-3" #72 daemon prio=5 os_prio=0 tid=0x00007f4cf4021000 nid=0x7441 waiting on condition [0x00007f4c83cfb000]
> 2020-06-04T11:34:52.0035891Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0036563Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0038036Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0039090Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0040263Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0041614Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0042956Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0044161Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0045427Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0046545Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0047480Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0047941Z
> 2020-06-04T11:34:52.0049160Z "mini-cluster-io-thread-13" #71 daemon prio=5 os_prio=0 tid=0x00007f4d000c2000 nid=0x7440 waiting on condition [0x00007f4c83dfc000]
> 2020-06-04T11:34:52.0050151Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0050796Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0052074Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0053149Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0054301Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0055639Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0056678Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0057821Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0058882Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0059783Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0060209Z
> 2020-06-04T11:34:52.0061450Z "jobmanager-future-thread-2" #70 daemon prio=5 os_prio=0 tid=0x00007f4d0802a800 nid=0x743f waiting on condition [0x00007f4c83efd000]
> 2020-06-04T11:34:52.0062567Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0063214Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0064577Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0065663Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0066842Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0068250Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0069584Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0070782Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0071836Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0072916Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0073789Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0074242Z
> 2020-06-04T11:34:52.0075755Z "FlinkCompletableFutureDelayScheduler-thread-1" #69 daemon prio=5 os_prio=0 tid=0x00007f4d08021000 nid=0x743e waiting on condition [0x00007f4c83ffe000]
> 2020-06-04T11:34:52.0076828Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0077536Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0078926Z - parking to wait for <0x000000008777e698> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0079998Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0081149Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0082514Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:52.0083830Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0085239Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0086384Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0087523Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0088428Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0088855Z
> 2020-06-04T11:34:52.0090097Z "mini-cluster-io-thread-12" #68 daemon prio=5 os_prio=0 tid=0x00007f4d0800d000 nid=0x743d waiting on condition [0x00007f4d115da000]
> 2020-06-04T11:34:52.0091065Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0091736Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0092994Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0094069Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0095322Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0096546Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0097702Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0098758Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0099837Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0100733Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0101161Z
> 2020-06-04T11:34:52.0102374Z "mini-cluster-io-thread-11" #67 daemon prio=5 os_prio=0 tid=0x00007f4d000b4800 nid=0x743c waiting on condition [0x00007f4d1196c000]
> 2020-06-04T11:34:52.0103496Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0104166Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0105557Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0106636Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0107863Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0109078Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0110111Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0111187Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0112263Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0113141Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0113590Z
> 2020-06-04T11:34:52.0114888Z "mini-cluster-io-thread-10" #64 daemon prio=5 os_prio=0 tid=0x00007f4cb000b000 nid=0x7439 waiting on condition [0x00007f4d10338000]
> 2020-06-04T11:34:52.0115908Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0116557Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0117907Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0125989Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0127262Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0128486Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0129522Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0130608Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0131667Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0132564Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0132994Z
> 2020-06-04T11:34:52.0134343Z "mini-cluster-io-thread-9" #63 daemon prio=5 os_prio=0 tid=0x00007f4cb0006800 nid=0x7438 waiting on condition [0x00007f4d10439000]
> 2020-06-04T11:34:52.0135607Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0136372Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0137749Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0138798Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0139975Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0141174Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0142229Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0143301Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0144361Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0145444Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0145909Z
> 2020-06-04T11:34:52.0147243Z "jobmanager-future-thread-1" #62 daemon prio=5 os_prio=0 tid=0x00007f4cb001c000 nid=0x7437 waiting on condition [0x00007f4d1053a000]
> 2020-06-04T11:34:52.0148220Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0148887Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0150135Z - parking to wait for <0x000000008797abe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0151379Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0152526Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0153885Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0155318Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0156536Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0157700Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0158762Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0159663Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0160089Z
> 2020-06-04T11:34:52.0161328Z "mini-cluster-io-thread-8" #60 daemon prio=5 os_prio=0 tid=0x00007f4cfc017800 nid=0x7435 waiting on condition [0x00007f4d1093c000]
> 2020-06-04T11:34:52.0162317Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0162964Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0164236Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0165367Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0166572Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0168520Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0169554Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0170631Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0171697Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0172596Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0173024Z
> 2020-06-04T11:34:52.0174262Z "mini-cluster-io-thread-7" #59 daemon prio=5 os_prio=0 tid=0x00007f4cfc016000 nid=0x7434 waiting on condition [0x00007f4d10a3d000]
> 2020-06-04T11:34:52.0175318Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0176020Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0177340Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0178416Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0179587Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0180777Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0181838Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0182888Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0183966Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0185038Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0185530Z
> 2020-06-04T11:34:52.0186770Z "mini-cluster-io-thread-6" #58 daemon prio=5 os_prio=0 tid=0x00007f4cfc014800 nid=0x7433 waiting on condition [0x00007f4d10b3e000]
> 2020-06-04T11:34:52.0187836Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0188506Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0189757Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0190825Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0191976Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0193303Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0194355Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0195678Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0197319Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0198196Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0198648Z
> 2020-06-04T11:34:52.0199892Z "mini-cluster-io-thread-5" #57 daemon prio=5 os_prio=0 tid=0x00007f4cb4003800 nid=0x7432 waiting on condition [0x00007f4d10c3f000]
> 2020-06-04T11:34:52.0200877Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0201523Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0202784Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0203856Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0205091Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0206342Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0207441Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0208512Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0209704Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0210593Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0211018Z
> 2020-06-04T11:34:52.0212247Z "mini-cluster-io-thread-4" #56 daemon prio=5 os_prio=0 tid=0x00007f4cc616e000 nid=0x7431 waiting on condition [0x00007f4d10d40000]
> 2020-06-04T11:34:52.0213243Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0213887Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0215292Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0216467Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0217738Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0218929Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0219990Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0221055Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0222114Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0223005Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0223439Z
> 2020-06-04T11:34:52.0224768Z "mini-cluster-io-thread-3" #55 daemon prio=5 os_prio=0 tid=0x00007f4cf4014800 nid=0x7430 waiting on condition [0x00007f4d10e41000]
> 2020-06-04T11:34:52.0225773Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0226441Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0227766Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0228837Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0230003Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0231194Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0232248Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0233294Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0234581Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0235493Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0235938Z
> 2020-06-04T11:34:52.0237207Z "mini-cluster-io-thread-2" #54 daemon prio=5 os_prio=0 tid=0x00007f4cc6141800 nid=0x742f waiting on condition [0x00007f4d10f42000]
> 2020-06-04T11:34:52.0238195Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0238860Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0240116Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0241187Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0242335Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0243547Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0244662Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0245767Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0246845Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0247775Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0248219Z
> 2020-06-04T11:34:52.0249480Z "Flink-DispatcherRestEndpoint-thread-1" #53 daemon prio=5 os_prio=0 tid=0x00007f4cc60c4800 nid=0x742e waiting on condition [0x00007f4d11043000]
> 2020-06-04T11:34:52.0250644Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:52.0251313Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0252604Z - parking to wait for <0x0000000087803238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0253674Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0254977Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:52.0256397Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:52.0257767Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0258964Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0260021Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0261102Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0261991Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0262415Z
> 2020-06-04T11:34:52.0263611Z "mini-cluster-io-thread-1" #52 daemon prio=5 os_prio=0 tid=0x00007f4cc60c3000 nid=0x742d waiting on condition [0x00007f4d11144000]
> 2020-06-04T11:34:52.0264670Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0265335Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0266619Z - parking to wait for <0x000000008797b270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0267750Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0268900Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0270115Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0271168Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0272219Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0273298Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0274314Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0274838Z
> 2020-06-04T11:34:52.0276095Z "flink-rest-server-netty-boss-thread-1" #51 daemon prio=5 os_prio=0 tid=0x00007f4cc6062000 nid=0x742c runnable [0x00007f4d11245000]
> 2020-06-04T11:34:52.0277062Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0277813Z at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 2020-06-04T11:34:52.0278653Z at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 2020-06-04T11:34:52.0279581Z at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> 2020-06-04T11:34:52.0280504Z at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-06-04T11:34:52.0281888Z - locked <0x00000000879040b0> (a org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySet)
> 2020-06-04T11:34:52.0283156Z - locked <0x00000000879040a0> (a java.util.Collections$UnmodifiableSet)
> 2020-06-04T11:34:52.0284265Z - locked <0x0000000087904058> (a sun.nio.ch.EPollSelectorImpl)
> 2020-06-04T11:34:52.0285156Z at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-06-04T11:34:52.0286383Z at org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> 2020-06-04T11:34:52.0287813Z at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)
> 2020-06-04T11:34:52.0288990Z at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)
> 2020-06-04T11:34:52.0290320Z at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> 2020-06-04T11:34:52.0291783Z at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2020-06-04T11:34:52.0292788Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0293213Z
> 2020-06-04T11:34:52.0294026Z "IOManager reader thread #1" #46 daemon prio=5 os_prio=0 tid=0x00007f4e711b9000 nid=0x7429 waiting on condition [0x00007f4d1206d000]
> 2020-06-04T11:34:52.0295102Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0295781Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0297076Z - parking to wait for <0x000000008797ba50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0298195Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0299365Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0300565Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0301714Z at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$ReaderThread.run(IOManagerAsync.java:354)
> 2020-06-04T11:34:52.0302413Z
> 2020-06-04T11:34:52.0303226Z "IOManager writer thread #1" #45 daemon prio=5 os_prio=0 tid=0x00007f4e711b7000 nid=0x7428 waiting on condition [0x00007f4d1216e000]
> 2020-06-04T11:34:52.0304216Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0304942Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0306264Z - parking to wait for <0x000000008777ee58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0307417Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0308735Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0309996Z at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0311166Z at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:460)
> 2020-06-04T11:34:52.0311865Z
> 2020-06-04T11:34:52.0312999Z "Timer-2" #43 daemon prio=5 os_prio=0 tid=0x00007f4e7117f000 nid=0x7427 in Object.wait() [0x00007f4d1226f000]
> 2020-06-04T11:34:52.0313971Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-06-04T11:34:52.0314967Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0316159Z - waiting on <0x000000008797bc58> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0316929Z at java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0318069Z - locked <0x000000008797bc58> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0318790Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0319253Z
> 2020-06-04T11:34:52.0320336Z "Timer-1" #41 daemon prio=5 os_prio=0 tid=0x00007f4e7117c800 nid=0x7426 in Object.wait() [0x00007f4d12370000]
> 2020-06-04T11:34:52.0321313Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-06-04T11:34:52.0322018Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0322990Z - waiting on <0x0000000087803a80> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0323768Z at java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0324863Z - locked <0x0000000087803a80> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0325652Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0326096Z
> 2020-06-04T11:34:52.0326861Z "BLOB Server listener at 41173" #37 daemon prio=5 os_prio=0 tid=0x00007f4e7117a000 nid=0x7425 runnable [0x00007f4d12471000]
> 2020-06-04T11:34:52.0327864Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0328567Z at java.net.PlainSocketImpl.socketAccept(Native Method)
> 2020-06-04T11:34:52.0329437Z at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> 2020-06-04T11:34:52.0330392Z at java.net.ServerSocket.implAccept(ServerSocket.java:560)
> 2020-06-04T11:34:52.0331352Z at java.net.ServerSocket.accept(ServerSocket.java:528)
> 2020-06-04T11:34:52.0332267Z at org.apache.flink.runtime.blob.BlobServer.run(BlobServer.java:262)
> 2020-06-04T11:34:52.0332818Z
> 2020-06-04T11:34:52.0333941Z "Timer-0" #38 daemon prio=5 os_prio=0 tid=0x00007f4e71160000 nid=0x7424 in Object.wait() [0x00007f4d12772000]
> 2020-06-04T11:34:52.0334980Z java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 2020-06-04T11:34:52.0335747Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0336733Z - waiting on <0x0000000087803a68> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0337565Z at java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0338608Z - locked <0x0000000087803a68> (a java.util.TaskQueue)
> 2020-06-04T11:34:52.0339327Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0339785Z
> 2020-06-04T11:34:52.0341535Z "flink-metrics-scheduler-1" #33 prio=5 os_prio=0 tid=0x00007f4e71126000 nid=0x7420 waiting on condition [0x00007f4d12b76000]
> 2020-06-04T11:34:52.0342549Z java.lang.Thread.State: TIMED_WAITING (sleeping)
> 2020-06-04T11:34:52.0343227Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:52.0344121Z at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
> 2020-06-04T11:34:52.0345384Z at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
> 2020-06-04T11:34:52.0346626Z at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
> 2020-06-04T11:34:52.0347630Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0348057Z
> 2020-06-04T11:34:52.0349204Z "flink-scheduler-1" #27 prio=5 os_prio=0 tid=0x00007f4e70d14800 nid=0x7418 waiting on condition [0x00007f4d13390000]
> 2020-06-04T11:34:52.0350172Z java.lang.Thread.State: TIMED_WAITING (sleeping)
> 2020-06-04T11:34:52.0350872Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:52.0351746Z at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
> 2020-06-04T11:34:52.0352929Z at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
> 2020-06-04T11:34:52.0354102Z at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
> 2020-06-04T11:34:52.0355209Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0355675Z
> 2020-06-04T11:34:52.0356453Z "process reaper" #24 daemon prio=10 os_prio=0 tid=0x00007f4d18048800 nid=0x7415 waiting on condition [0x00007f4d13dce000]
> 2020-06-04T11:34:52.0357656Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:52.0358349Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0359574Z - parking to wait for <0x0000000080c55a08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> 2020-06-04T11:34:52.0360584Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0361681Z at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 2020-06-04T11:34:52.0362832Z at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> 2020-06-04T11:34:52.0363905Z at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> 2020-06-04T11:34:52.0365103Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> 2020-06-04T11:34:52.0366197Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0367350Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0368223Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0368669Z
> 2020-06-04T11:34:52.0369901Z "surefire-forkedjvm-ping-30s" #23 daemon prio=5 os_prio=0 tid=0x00007f4e703f6000 nid=0x7412 waiting on condition [0x00007f4d343c2000]
> 2020-06-04T11:34:52.0370924Z java.lang.Thread.State: TIMED_WAITING (parking)
> 2020-06-04T11:34:52.0371592Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0372853Z - parking to wait for <0x0000000080c79c40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0374047Z at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0375351Z at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:52.0376773Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:52.0378153Z at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0379351Z at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0380402Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0381480Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0382367Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0382797Z
> 2020-06-04T11:34:52.0383991Z "surefire-forkedjvm-command-thread" #22 daemon prio=5 os_prio=0 tid=0x00007f4e703c4000 nid=0x7411 runnable [0x00007f4d346cd000]
> 2020-06-04T11:34:52.0385010Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0385725Z at java.io.FileInputStream.readBytes(Native Method)
> 2020-06-04T11:34:52.0386500Z at java.io.FileInputStream.read(FileInputStream.java:255)
> 2020-06-04T11:34:52.0387475Z at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> 2020-06-04T11:34:52.0388394Z at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> 2020-06-04T11:34:52.0389548Z - locked <0x0000000080c9fff8> (a java.io.BufferedInputStream)
> 2020-06-04T11:34:52.0390386Z at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 2020-06-04T11:34:52.0391401Z at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
> 2020-06-04T11:34:52.0392588Z at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
> 2020-06-04T11:34:52.0393518Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0393960Z
> 2020-06-04T11:34:52.0394775Z "Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007f4e702d5000 nid=0x740f runnable [0x0000000000000000]
> 2020-06-04T11:34:52.0395757Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0396158Z
> 2020-06-04T11:34:52.0396945Z "C1 CompilerThread14" #20 daemon prio=9 os_prio=0 tid=0x00007f4e702c8000 nid=0x740e waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0398088Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0398505Z
> 2020-06-04T11:34:52.0399271Z "C1 CompilerThread13" #19 daemon prio=9 os_prio=0 tid=0x00007f4e702c5800 nid=0x740d waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0400201Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0400594Z
> 2020-06-04T11:34:52.0401379Z "C1 CompilerThread12" #18 daemon prio=9 os_prio=0 tid=0x00007f4e702c3800 nid=0x740c waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0402290Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0402685Z
> 2020-06-04T11:34:52.0403469Z "C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007f4e702c1800 nid=0x740b waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0404393Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0404872Z
> 2020-06-04T11:34:52.0405666Z "C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007f4e702c0000 nid=0x740a waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0406601Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0406994Z
> 2020-06-04T11:34:52.0407850Z "C2 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007f4e702bd000 nid=0x7409 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0408779Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0409172Z
> 2020-06-04T11:34:52.0409935Z "C2 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007f4e702bb000 nid=0x7408 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0410862Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0411376Z
> 2020-06-04T11:34:52.0412166Z "C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007f4e702b9000 nid=0x7407 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0413076Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0413488Z
> 2020-06-04T11:34:52.0414247Z "C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007f4e702b7000 nid=0x7406 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0415279Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0415709Z
> 2020-06-04T11:34:52.0416490Z "C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007f4e702b5000 nid=0x7405 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0417456Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0417872Z
> 2020-06-04T11:34:52.0418634Z "C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007f4e702b3000 nid=0x7404 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0419554Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0419952Z
> 2020-06-04T11:34:52.0420729Z "C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f4e702a9000 nid=0x7403 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0421635Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0422055Z
> 2020-06-04T11:34:52.0422815Z "C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f4e702a6800 nid=0x7402 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0423747Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0424141Z
> 2020-06-04T11:34:52.0424985Z "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f4e702a4800 nid=0x7401 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0425921Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0426339Z
> 2020-06-04T11:34:52.0427094Z "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f4e702a2800 nid=0x7400 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0428062Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0428457Z
> 2020-06-04T11:34:52.0429191Z "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f4e702a0800 nid=0x73ff runnable [0x0000000000000000]
> 2020-06-04T11:34:52.0430050Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0430443Z
> 2020-06-04T11:34:52.0431290Z "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f4e7029f000 nid=0x73fe waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0432508Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0432969Z
> 2020-06-04T11:34:52.0433783Z "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4e7026e800 nid=0x73fd in Object.wait() [0x00007f4d7d0b2000]
> 2020-06-04T11:34:52.0434997Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-06-04T11:34:52.0435827Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0437241Z - waiting on <0x0000000080c56688> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:52.0438175Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-06-04T11:34:52.0439453Z - locked <0x0000000080c56688> (a java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:52.0440348Z at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-06-04T11:34:52.0441259Z at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> 2020-06-04T11:34:52.0441825Z
> 2020-06-04T11:34:52.0442576Z "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4e7026a000 nid=0x73fc in Object.wait() [0x00007f4d7d1b3000]
> 2020-06-04T11:34:52.0443567Z java.lang.Thread.State: WAITING (on object monitor)
> 2020-06-04T11:34:52.0444269Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0445010Z at java.lang.Object.wait(Object.java:502)
> 2020-06-04T11:34:52.0445862Z at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> 2020-06-04T11:34:52.0447007Z - locked <0x0000000080c56678> (a java.lang.ref.Reference$Lock)
> 2020-06-04T11:34:52.0447959Z at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> 2020-06-04T11:34:52.0448509Z
> 2020-06-04T11:34:52.0449218Z "main" #1 prio=5 os_prio=0 tid=0x00007f4e7000b800 nid=0x73c4 waiting on condition [0x00007f4e768fd000]
> 2020-06-04T11:34:52.0450228Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0450892Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0452052Z - parking to wait for <0x000000008757aa28> (a java.util.concurrent.CompletableFuture$Signaller)
> 2020-06-04T11:34:52.0453050Z at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0454097Z at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2020-06-04T11:34:52.0455234Z at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> 2020-06-04T11:34:52.0456299Z at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2020-06-04T11:34:52.0457383Z at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-06-04T11:34:52.0458470Z at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:671)
> 2020-06-04T11:34:52.0459639Z at org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:81)
> 2020-06-04T11:34:52.0460933Z at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1685)
> 2020-06-04T11:34:52.0462322Z at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1667)
> 2020-06-04T11:34:52.0463724Z at org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase.executeAndVerify(CoordinatedSourceITCase.java:84)
> 2020-06-04T11:34:52.0465237Z at org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase.testMultipleSources(CoordinatedSourceITCase.java:68)
> 2020-06-04T11:34:52.0466388Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-06-04T11:34:52.0467359Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-06-04T11:34:52.0468461Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-06-04T11:34:52.0469422Z at java.lang.reflect.Method.invoke(Method.java:498)
> 2020-06-04T11:34:52.0470390Z at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-06-04T11:34:52.0471490Z at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-06-04T11:34:52.0472601Z at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-06-04T11:34:52.0473807Z at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-06-04T11:34:52.0474938Z at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-06-04T11:34:52.0475922Z at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T11:34:52.0476784Z at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-06-04T11:34:52.0477853Z at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-06-04T11:34:52.0478931Z at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-06-04T11:34:52.0479942Z at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-06-04T11:34:52.0480862Z at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-06-04T11:34:52.0481788Z at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-06-04T11:34:52.0482735Z at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-06-04T11:34:52.0483662Z at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-06-04T11:34:52.0484764Z at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T11:34:52.0485851Z at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T11:34:52.0486750Z at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T11:34:52.0487677Z at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-06-04T11:34:52.0488648Z at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-06-04T11:34:52.0489900Z at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-06-04T11:34:52.0491041Z at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-06-04T11:34:52.0492159Z at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-06-04T11:34:52.0493333Z at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-06-04T11:34:52.0494619Z at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-06-04T11:34:52.0495855Z at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2020-06-04T11:34:52.0496879Z at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2020-06-04T11:34:52.0497557Z
> 2020-06-04T11:34:52.0498119Z "VM Thread" os_prio=0 tid=0x00007f4e70260800 nid=0x73fb runnable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)