You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Shai Kaplan <Sh...@microsoft.com> on 2017/02/21 13:47:14 UTC

Flink checkpointing gets stuck

Hi.
I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After some running time (minutes-hours) Flink fails to save checkpoints, and stops processing records (I'm not sure if the checkpointing failure is the cause of the problem or just a symptom).
After several checkpoints that take some seconds each, they start failing due to 30 minutes timeout.
When I restart one of the Task Manager services (just to get the job restarted), the job is recovered from the last successful checkpoint (the state size continues to grow, so it's probably not the reason for the failure), advances somewhat, saves some more checkpoints, and then enters the failing state again.
One of the times it happened, the first failed checkpoint failed due to "Checkpoint Coordinator is suspending.", so it might be an indicator for the cause of the problem, but looking into Flink's code I can't see how a running job could get to this state.
I am using RocksDB for state, and the state is saved to Azure Blob Store, using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
Any ideas? Possibly a bug in Flink or RocksDB?

Re: Flink checkpointing gets stuck

Posted by vinay patil <vi...@gmail.com>.

Hi Shai,

I checked online that Azure DS5_v2 has SSD for storage, why don't you try
to use FLASH_SSD_OPTIMIZED option

In my case as well the stream was getting stuck for few minutes, my
checkpoint duration is 6secs and minimumPauseIntervalBetweenCheckpoints is
5secs

https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/large_state_tuning.html

I think if the writes to RocksDB are blocked then the stream can block for
certain interval
https://github.com/facebook/rocksdb/wiki/Write-Stalls

First try with FLASH_SSD_OPTIMIZED option, and don't give unnecessary high
heap memory to TM as rocksDB also uses physical memory




Regards,
Vinay Patil

On Tue, Feb 21, 2017 at 8:03 PM, Shai Kaplan [via Apache Flink User Mailing
List archive.] <ml...@n4.nabble.com> wrote:

> Hi Vinay.
>
>
>
> I couldn't understand from the thread, what configuration solved your
> problem?
>
>
>
> I'm using the default predefined option. Perhaps it's not the best
> configuration for my setting (I'm using Azure DS5_v2 machines), I honestly
> haven't given much thought to that particular detail, but I think it should
> only affect the performance, not make the job totally stuck.
>
>
>
> Thanks.
>
>
>
> *From:* vinay patil [mailto:[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=11780&i=0>]
> *Sent:* Tuesday, February 21, 2017 3:58 PM
> *To:* [hidden email] <http:///user/SendEmail.jtp?type=node&node=11780&i=1>
> *Subject:* Re: Flink checkpointing gets stuck
>
>
>
> Hi Shai,
>
> I was facing similar issue , however now the stream is not stuck in
> between.
>
> you can refer this thread for the configurations I have done :
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-
> Checkpointing-with-RocksDB-as-statebackend-td11752.html
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FRe-Checkpointing-with-RocksDB-as-statebackend-td11752.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=z0YAi2n6itetqIfkD6tuOpHKQY0qbOLNUuAoYiQEWak%3D&reserved=0>
>
> What is the configuration on which you running the job ?
> What is the RocksDB predefined option you are using ?
>
>
> Regards,
>
> Vinay Patil
>
>
>
> On Tue, Feb 21, 2017 at 7:13 PM, Shai Kaplan [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=11778&i=0>> wrote:
>
> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After
> some running time (minutes-hours) Flink fails to save checkpoints, and
> stops processing records (I'm not sure if the checkpointing failure is the
> cause of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start failing
> due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job
> restarted), the job is recovered from the last successful checkpoint (the
> state size continues to grow, so it's probably not the reason for the
> failure), advances somewhat, saves some more checkpoints, and then enters
> the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due to
> "Checkpoint Coordinator is suspending.", so it might be an indicator for
> the cause of the problem, but looking into Flink's code I can't see how a
> running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob Store,
> using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?
>
>
> ------------------------------
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-
> checkpointing-gets-stuck-tp11776.html
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FFlink-checkpointing-gets-stuck-tp11776.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=Qt7qCSOvhSkzQA1y9ze13UqEotuWt0yKSQJ9gIV1DW8%3D&reserved=0>
>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden
> email] <http:///user/SendEmail.jtp?type=node&node=11778&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here.
> NAML
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=JdgLltimPhln4llGkLOpTCvHKy2GFVUC%2BuoM5gZOH4w%3D&reserved=0>
>
>
>
>
> ------------------------------
>
> View this message in context: Re: Flink checkpointing gets stuck
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FFlink-checkpointing-gets-stuck-tp11776p11778.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326834807&sdata=vtsd7KXC3G5zn3ZmCEyo0RYi16TJjrrzj%2FG8a%2BPBECs%3D&reserved=0>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2F&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326834807&sdata=JjFgdLMaCzZ9FcQ992QUZtnP%2BjxAZghzA7g05nBurLU%3D&reserved=0>
> at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-
> checkpointing-gets-stuck-tp11776p11780.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-checkpointing-gets-stuck-tp11776p11783.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

RE: Flink checkpointing gets stuck

Posted by Shai Kaplan <Sh...@microsoft.com>.

Hi Vinay.

I couldn't understand from the thread, what configuration solved your problem?

I'm using the default predefined option. Perhaps it's not the best configuration for my setting (I'm using Azure DS5_v2 machines), I honestly haven't given much thought to that particular detail, but I think it should only affect the performance, not make the job totally stuck.

Thanks.

From: vinay patil [mailto:vinay18.patil@gmail.com]
Sent: Tuesday, February 21, 2017 3:58 PM
To: user@flink.apache.org
Subject: Re: Flink checkpointing gets stuck

Hi Shai,

I was facing similar issue , however now the stream is not stuck in between.
you can refer this thread for the configurations I have done : http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-td11752.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FRe-Checkpointing-with-RocksDB-as-statebackend-td11752.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=z0YAi2n6itetqIfkD6tuOpHKQY0qbOLNUuAoYiQEWak%3D&reserved=0>

What is the configuration on which you running the job ?
What is the RocksDB predefined option you are using ?


Regards,
Vinay Patil

On Tue, Feb 21, 2017 at 7:13 PM, Shai Kaplan [via Apache Flink User Mailing List archive.] <[hidden email]</user/SendEmail.jtp?type=node&node=11778&i=0>> wrote:
Hi.
I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After some running time (minutes-hours) Flink fails to save checkpoints, and stops processing records (I'm not sure if the checkpointing failure is the cause of the problem or just a symptom).
After several checkpoints that take some seconds each, they start failing due to 30 minutes timeout.
When I restart one of the Task Manager services (just to get the job restarted), the job is recovered from the last successful checkpoint (the state size continues to grow, so it's probably not the reason for the failure), advances somewhat, saves some more checkpoints, and then enters the failing state again.
One of the times it happened, the first failed checkpoint failed due to "Checkpoint Coordinator is suspending.", so it might be an indicator for the cause of the problem, but looking into Flink's code I can't see how a running job could get to this state.
I am using RocksDB for state, and the state is saved to Azure Blob Store, using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
Any ideas? Possibly a bug in Flink or RocksDB?

________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-checkpointing-gets-stuck-tp11776.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FFlink-checkpointing-gets-stuck-tp11776.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=Qt7qCSOvhSkzQA1y9ze13UqEotuWt0yKSQJ9gIV1DW8%3D&reserved=0>
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]</user/SendEmail.jtp?type=node&node=11778&i=1>
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326824798&sdata=JdgLltimPhln4llGkLOpTCvHKy2GFVUC%2BuoM5gZOH4w%3D&reserved=0>


________________________________
View this message in context: Re: Flink checkpointing gets stuck<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FFlink-checkpointing-gets-stuck-tp11776p11778.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326834807&sdata=vtsd7KXC3G5zn3ZmCEyo0RYi16TJjrrzj%2FG8a%2BPBECs%3D&reserved=0>
Sent from the Apache Flink User Mailing List archive. mailing list archive<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2F&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C70dc9d483010493b7fd308d45a623b0f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232825326834807&sdata=JjFgdLMaCzZ9FcQ992QUZtnP%2BjxAZghzA7g05nBurLU%3D&reserved=0> at Nabble.com.

Re: Flink checkpointing gets stuck

Posted by vinay patil <vi...@gmail.com>.

Hi Shai,

I was facing similar issue , however now the stream is not stuck in between.

you can refer this thread for the configurations I have done :
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-td11752.html

What is the configuration on which you running the job ?
What is the RocksDB predefined option you are using ?



Regards,
Vinay Patil

On Tue, Feb 21, 2017 at 7:13 PM, Shai Kaplan [via Apache Flink User Mailing
List archive.] <ml...@n4.nabble.com> wrote:

> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After
> some running time (minutes-hours) Flink fails to save checkpoints, and
> stops processing records (I'm not sure if the checkpointing failure is the
> cause of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start failing
> due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job
> restarted), the job is recovered from the last successful checkpoint (the
> state size continues to grow, so it's probably not the reason for the
> failure), advances somewhat, saves some more checkpoints, and then enters
> the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due to
> "Checkpoint Coordinator is suspending.", so it might be an indicator for
> the cause of the problem, but looking into Flink's code I can't see how a
> running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob Store,
> using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-
> checkpointing-gets-stuck-tp11776.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-checkpointing-gets-stuck-tp11776p11778.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Flink checkpointing gets stuck

Posted by Ufuk Celebi <uc...@apache.org>.

Added this JIRA https://issues.apache.org/jira/browse/FLINK-5993 to
track this. Would be great to comment there if you have any other
issues that should be covered in a Azure deployment section.


On Tue, Mar 7, 2017 at 7:10 PM, Stephan Ewen <se...@apache.org> wrote:
> Great to hear it!
>
> What do you think about adding a section to the Flink docs about deployment
> on Azure (there is already AWS and GCE, so Azure would make the
> cloud-trinity complete) that explains how to set this up and avoid such
> pitfalls.
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/
>
> If there is anything you think can be improved in the dependencies to help
> there, please let us know!
>
>
> On Tue, Mar 7, 2017 at 5:31 PM, Avihai Berkovitz
> <Av...@microsoft.com> wrote:
>>
>> For anyone seeing this thread in the future, we managed to solve the
>> issue. The problem was in the Azure storage SDK.
>>
>> Flink is using Hadoop 2.7, so we used version 2.7.3 of the Hadoop-azure
>> package. This package uses version 2.0.0 of the azure-storage package, dated
>> from 2014. It has several bugs that were since fixed, specifically one where
>> the socket timeout was infinite. We updated this package to version 5.0.0
>> and everything is working smoothly now.
>>
>>
>>
>> From: Stephan Ewen [mailto:sewen@apache.org]
>> Sent: Sunday, February 26, 2017 4:47 PM
>> To: user@flink.apache.org
>> Subject: Re: Flink checkpointing gets stuck
>>
>>
>>
>> Thanks!
>>
>>
>>
>> This looks like a bigger example, involving MongoDB, etc.
>>
>> Are you able to reproduce this issue with a smaller example?
>>
>>
>>
>> It would also help to understand the problem better if we knew the
>> topology a bit better.
>>
>> The stack traces look like "phase 1&2" want to send data (but are back
>> pressured) and "phase 3&4&5" wait for input data.
>>
>>
>>
>> Stephan
>>
>>
>>
>>
>>
>> On Sun, Feb 26, 2017 at 12:30 PM, Shai Kaplan <Sh...@microsoft.com>
>> wrote:
>>
>> Running jstack on one of the Task Managers:
>>
>>
>>
>> 2017-02-26 10:06:27
>>
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed
>> mode):
>>
>>
>>
>> "Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000
>> nid=0xe692 waiting on condition [0x0000000000000000]
>>
>>    java.lang.Thread.State: RUNNABLE
>>
>>
>>
>> "Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0
>> tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d000620> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0
>> tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd7e70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07cdde8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e0003e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8008> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e000730> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07cdf80> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e000a78> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd81a0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0
>> tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d000968> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0
>> tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07ce118> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0
>> tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8338> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0
>> tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d000cb0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0
>> tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07ce2b0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0
>> tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd84d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0
>> tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e000dc0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0
>> tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d000ff8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0
>> tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07ce448> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0
>> tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8668> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0
>> tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d001340> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0
>> tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e001108> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0
>> tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07ce5e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0
>> tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8800> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0
>> tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e001450> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0
>> tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d001688> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0
>> tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e001798> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0
>> tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d0019d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0
>> tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8998> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0
>> tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3da07ce778> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0
>> tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9e001ae0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0
>> tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9d001d18> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0
>> tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.park(Native Method)
>>
>>               - parking to wait for  <0x00007f3d9fbd8b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>               at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>
>>               at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>
>>               at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>
>>               at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>>               at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> "Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0
>> tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
>>
>>    java.lang.Thread.State: WAITING (parking)
>>
>>               at sun.misc.Unsafe.pa
>>
>> ...
>>
>> [Message clipped]
>>
>>
>
>

Re: Flink checkpointing gets stuck

Posted by Stephan Ewen <se...@apache.org>.

Great to hear it!

What do you think about adding a section to the Flink docs about deployment
on Azure (there is already AWS and GCE, so Azure would make the
cloud-trinity complete) that explains how to set this up and avoid such
pitfalls.

https://ci.apache.org/projects/flink/flink-docs-release-1.2/

If there is anything you think can be improved in the dependencies to help
there, please let us know!


On Tue, Mar 7, 2017 at 5:31 PM, Avihai Berkovitz <
Avihai.Berkovitz@microsoft.com> wrote:

> For anyone seeing this thread in the future, we managed to solve the
> issue. The problem was in the Azure storage SDK.
>
> Flink is using Hadoop 2.7, so we used version 2.7.3 of the Hadoop-azure
> package. This package uses version 2.0.0 of the azure-storage package,
> dated from 2014. It has several bugs that were since fixed, specifically
> one where the socket timeout was infinite. We updated this package to
> version 5.0.0 and everything is working smoothly now.
>
>
>
> *From:* Stephan Ewen [mailto:sewen@apache.org]
> *Sent:* Sunday, February 26, 2017 4:47 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink checkpointing gets stuck
>
>
>
> Thanks!
>
>
>
> This looks like a bigger example, involving MongoDB, etc.
>
> Are you able to reproduce this issue with a smaller example?
>
>
>
> It would also help to understand the problem better if we knew the
> topology a bit better.
>
> The stack traces look like "phase 1&2" want to send data (but are back
> pressured) and "phase 3&4&5" wait for input data.
>
>
>
> Stephan
>
>
>
>
>
> On Sun, Feb 26, 2017 at 12:30 PM, Shai Kaplan <Sh...@microsoft.com>
> wrote:
>
> Running jstack on one of the Task Managers:
>
>
>
> 2017-02-26 10:06:27
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):
>
>
>
> "Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000
> nid=0xe692 waiting on condition [0x0000000000000000]
>
>    java.lang.Thread.State: RUNNABLE
>
>
>
> "Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0
> tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000620> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0
> tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd7e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0
> tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdde8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0
> tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e0003e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0
> tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0
> tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000730> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0
> tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdf80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0
> tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000a78> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0
> tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd81a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0
> tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000968> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0
> tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0
> tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8338> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0
> tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000cb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0
> tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce2b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0
> tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd84d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0
> tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000dc0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0
> tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000ff8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0
> tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce448> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0
> tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8668> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0
> tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001340> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0
> tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0
> tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce5e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0
> tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0
> tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001450> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0
> tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001688> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0
> tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0
> tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d0019d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0
> tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8998> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0
> tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce778> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0
> tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001ae0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0
> tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001d18> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0
> tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0
> tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.pa
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsun.misc.Unsafe.pa&data=02%7C01%7CAvihai.Berkovitz%40microsoft.com%7Ca43b760ecbc0486ebaf008d45e565fdb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636237172464130817&sdata=lSZZtCUQk8vQyHPap889W1DGCpTHi7wPFxi3VhhNFsw%3D&reserved=0>
>
> ...
>
> [Message clipped]
>
>
>

RE: Flink checkpointing gets stuck

Posted by Avihai Berkovitz <Av...@microsoft.com>.

For anyone seeing this thread in the future, we managed to solve the issue. The problem was in the Azure storage SDK.
Flink is using Hadoop 2.7, so we used version 2.7.3 of the Hadoop-azure package. This package uses version 2.0.0 of the azure-storage package, dated from 2014. It has several bugs that were since fixed, specifically one where the socket timeout was infinite. We updated this package to version 5.0.0 and everything is working smoothly now.

From: Stephan Ewen [mailto:sewen@apache.org]
Sent: Sunday, February 26, 2017 4:47 PM
To: user@flink.apache.org
Subject: Re: Flink checkpointing gets stuck

Thanks!

This looks like a bigger example, involving MongoDB, etc.
Are you able to reproduce this issue with a smaller example?

It would also help to understand the problem better if we knew the topology a bit better.
The stack traces look like "phase 1&2" want to send data (but are back pressured) and "phase 3&4&5" wait for input data.

Stephan


On Sun, Feb 26, 2017 at 12:30 PM, Shai Kaplan <Sh...@microsoft.com>> wrote:
Running jstack on one of the Task Managers:

2017-02-26 10:06:27
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):

"Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000 nid=0xe692 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0 tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000620> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0 tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd7e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0 tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cdde8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0 tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e0003e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0 tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0 tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000730> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0 tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cdf80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0 tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0 tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd81a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0 tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0 tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0 tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0 tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0 tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce2b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0 tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd84d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0 tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000dc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0 tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000ff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0 tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce448> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0 tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8668> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0 tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0 tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001108> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0 tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce5e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0 tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0 tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001450> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0 tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001688> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0 tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001798> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0 tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d0019d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0 tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0 tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce778> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0 tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001ae0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0 tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0 tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0 tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.pa<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsun.misc.Unsafe.pa&data=02%7C01%7CAvihai.Berkovitz%40microsoft.com%7Ca43b760ecbc0486ebaf008d45e565fdb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636237172464130817&sdata=lSZZtCUQk8vQyHPap889W1DGCpTHi7wPFxi3VhhNFsw%3D&reserved=0>
...

[Message clipped]

Re: Flink checkpointing gets stuck

Posted by Stephan Ewen <se...@apache.org>.

Thanks!

This looks like a bigger example, involving MongoDB, etc.
Are you able to reproduce this issue with a smaller example?

It would also help to understand the problem better if we knew the topology
a bit better.
The stack traces look like "phase 1&2" want to send data (but are back
pressured) and "phase 3&4&5" wait for input data.

Stephan


On Sun, Feb 26, 2017 at 12:30 PM, Shai Kaplan <Sh...@microsoft.com>
wrote:

> Running jstack on one of the Task Managers:
>
>
>
> 2017-02-26 10:06:27
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):
>
>
>
> "Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000
> nid=0xe692 waiting on condition [0x0000000000000000]
>
>    java.lang.Thread.State: RUNNABLE
>
>
>
> "Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0
> tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000620> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0
> tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd7e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0
> tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdde8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0
> tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e0003e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0
> tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0
> tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000730> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0
> tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdf80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0
> tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000a78> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0
> tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd81a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0
> tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000968> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0
> tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0
> tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8338> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0
> tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000cb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0
> tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce2b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0
> tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd84d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0
> tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000dc0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0
> tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000ff8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0
> tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce448> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0
> tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8668> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0
> tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001340> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0
> tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0
> tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce5e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0
> tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0
> tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001450> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0
> tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001688> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0
> tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0
> tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d0019d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0
> tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8998> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0
> tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce778> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0
> tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001ae0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0
> tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001d18> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0
> tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0
> tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.pa
> ...
>
> [Message clipped]

RE: Flink checkpointing gets stuck

Posted by Shai Kaplan <Sh...@microsoft.com>.

Running jstack on one of the Task Managers:

2017-02-26 10:06:27
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):

"Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000 nid=0xe692 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0 tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000620> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0 tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd7e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0 tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cdde8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0 tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e0003e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0 tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0 tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000730> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0 tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cdf80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0 tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0 tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd81a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0 tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0 tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0 tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0 tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0 tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce2b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0 tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd84d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0 tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e000dc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0 tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d000ff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0 tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce448> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0 tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8668> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0 tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0 tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001108> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0 tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce5e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0 tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0 tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001450> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0 tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001688> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0 tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001798> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0 tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d0019d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0 tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0 tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce778> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0 tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001ae0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0 tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d001d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0 tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0 tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e001e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (31/48)" #2304 daemon prio=5 os_prio=0 tid=0x00007f3b9423d800 nid=0xb0b4 waiting on condition [0x00007f3af10d3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ce910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (30/48)" #2303 daemon prio=5 os_prio=0 tid=0x00007f3b9423c000 nid=0xb0b3 waiting on condition [0x00007f3af18db000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d002060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (29/48)" #2302 daemon prio=5 os_prio=0 tid=0x00007f3b9423a800 nid=0xb0b2 waiting on condition [0x00007f3af1bde000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e002170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (28/48)" #2301 daemon prio=5 os_prio=0 tid=0x00007f3b94239000 nid=0xb0b1 waiting on condition [0x00007f3af3b44000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8cc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (27/48)" #2300 daemon prio=5 os_prio=0 tid=0x00007f3b94237800 nid=0xb0b0 waiting on condition [0x00007f3af3c45000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07ceaa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (26/48)" #2299 daemon prio=5 os_prio=0 tid=0x00007f3b94236000 nid=0xb0af waiting on condition [0x00007f3af515a000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d0023a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (25/48)" #2298 daemon prio=5 os_prio=0 tid=0x00007f3b94234800 nid=0xb0ae waiting on condition [0x00007f3af9a60000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e0024b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (24/48)" #2297 daemon prio=5 os_prio=0 tid=0x00007f3b94233800 nid=0xb0ad waiting on condition [0x00007f3b018b1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8e60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (23/48)" #2296 daemon prio=5 os_prio=0 tid=0x00007f3b94232000 nid=0xb0ac waiting on condition [0x00007f3af11d4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e002800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (22/48)" #2295 daemon prio=5 os_prio=0 tid=0x00007f3b94230800 nid=0xb0ab waiting on condition [0x00007f3af26e9000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d0026f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (21/48)" #2294 daemon prio=5 os_prio=0 tid=0x00007f3b9422f000 nid=0xb0aa waiting on condition [0x00007f3ae3bfa000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cec40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (20/48)" #2293 daemon prio=5 os_prio=0 tid=0x00007f3b9422e800 nid=0xb0a9 waiting on condition [0x00007f3ae0ac9000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e002b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (19/48)" #2292 daemon prio=5 os_prio=0 tid=0x00007f3b94224800 nid=0xb0a8 waiting on condition [0x00007f3ae08c7000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd8ff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (18/48)" #2291 daemon prio=5 os_prio=0 tid=0x00007f3b94223000 nid=0xb0a7 waiting on condition [0x00007f3af16d9000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d002a38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 3 (17/48)" #2290 daemon prio=5 os_prio=0 tid=0x00007f3b94221800 nid=0xb0a6 waiting on condition [0x00007f3af25e8000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cedd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (32/48)" #2289 daemon prio=5 os_prio=0 tid=0x00007f3b94220000 nid=0xb0a5 waiting on condition [0x00007f3af3841000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e002e90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (31/48)" #2288 daemon prio=5 os_prio=0 tid=0x00007f3b9421f000 nid=0xb0a4 waiting on condition [0x00007f3ae0bca000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d002d80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (30/48)" #2287 daemon prio=5 os_prio=0 tid=0x00007f3b9421d800 nid=0xb0a3 waiting on condition [0x00007f3af0dd0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd9190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (29/48)" #2286 daemon prio=5 os_prio=0 tid=0x00007f3b9421c000 nid=0xb0a2 waiting on condition [0x00007f3af17da000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e0031d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (28/48)" #2285 daemon prio=5 os_prio=0 tid=0x00007f3b9421a800 nid=0xb0a1 waiting on condition [0x00007f3af424b000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cef70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (27/48)" #2284 daemon prio=5 os_prio=0 tid=0x00007f3b94219800 nid=0xb0a0 waiting on condition [0x00007f3af3a43000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d0030c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (26/48)" #2283 daemon prio=5 os_prio=0 tid=0x00007f3b942c9000 nid=0xb09f waiting on condition [0x00007f3ae11d0000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e003520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (25/48)" #2282 daemon prio=5 os_prio=0 tid=0x00007f3b942c8000 nid=0xb09e waiting on condition [0x00007f3af12d5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd9328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (24/48)" #2281 daemon prio=5 os_prio=0 tid=0x00007f3b942c6800 nid=0xb09d waiting on condition [0x00007f3af22e5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cf108> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (23/48)" #2280 daemon prio=5 os_prio=0 tid=0x00007f3b942c5000 nid=0xb09c waiting on condition [0x00007f3ae10cf000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d003410> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (22/48)" #2279 daemon prio=5 os_prio=0 tid=0x00007f3b942c3800 nid=0xb09b waiting on condition [0x00007f3af0bce000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e003868> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (21/48)" #2278 daemon prio=5 os_prio=0 tid=0x00007f3b942c2800 nid=0xb09a waiting on condition [0x00007f3ae37f6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd94c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (20/48)" #2277 daemon prio=5 os_prio=0 tid=0x00007f3b942c1000 nid=0xb099 waiting on condition [0x00007f3af14d7000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e003bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (19/48)" #2276 daemon prio=5 os_prio=0 tid=0x00007f3b942bf800 nid=0xb098 waiting on condition [0x00007f3ae36f5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d003758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (18/48)" #2275 daemon prio=5 os_prio=0 tid=0x00007f3b942bf000 nid=0xb097 waiting on condition [0x00007f3af13d6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cf2a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 2 (17/48)" #2274 daemon prio=5 os_prio=0 tid=0x00007f3b94262800 nid=0xb096 waiting on condition [0x00007f3af19dc000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e003ef8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (48/48)" #2273 daemon prio=5 os_prio=0 tid=0x00007f3b94261000 nid=0xb095 waiting on condition [0x00007f3af07ca000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd9658> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (46/48)" #2272 daemon prio=5 os_prio=0 tid=0x00007f3b9425f800 nid=0xb094 waiting on condition [0x00007f3af3740000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d003aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (44/48)" #2271 daemon prio=5 os_prio=0 tid=0x00007f3b9425e000 nid=0xb093 waiting on condition [0x00007f3af1cdf000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cf438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (42/48)" #2270 daemon prio=5 os_prio=0 tid=0x00007f3b94269800 nid=0xb092 waiting on condition [0x00007f3af1ee1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e004240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (40/48)" #2269 daemon prio=5 os_prio=0 tid=0x00007f3b94268800 nid=0xb091 waiting on condition [0x00007f3ae05c4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d003de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (38/48)" #2268 daemon prio=5 os_prio=0 tid=0x00007f3b94267000 nid=0xb090 waiting on condition [0x00007f3af3942000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd97f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (36/48)" #2267 daemon prio=5 os_prio=0 tid=0x00007f3b94265800 nid=0xb08f waiting on condition [0x00007f3af4049000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e004588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (34/48)" #2266 daemon prio=5 os_prio=0 tid=0x00007f3b94264000 nid=0xb08e waiting on condition [0x00007f3af565f000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cf5d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (32/48)" #2265 daemon prio=5 os_prio=0 tid=0x00007f3b94274800 nid=0xb08d waiting on condition [0x00007f3ae07c6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d004130> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (30/48)" #2264 daemon prio=5 os_prio=0 tid=0x00007f3b94273800 nid=0xb08c waiting on condition [0x00007f3af9c62000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e0048d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (28/48)" #2263 daemon prio=5 os_prio=0 tid=0x00007f3b94273000 nid=0xb08b waiting on condition [0x00007f3af28eb000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd9988> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (26/48)" #2262 daemon prio=5 os_prio=0 tid=0x00007f3b94272800 nid=0xb08a waiting on condition [0x00007f3af0ed1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3da07cf768> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (24/48)" #2261 daemon prio=5 os_prio=0 tid=0x00007f3b94271800 nid=0xb089 waiting on condition [0x00007f3af01c4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9d004478> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (22/48)" #2260 daemon prio=5 os_prio=0 tid=0x00007f3b94271000 nid=0xb088 waiting on condition [0x00007f3af4f58000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e004c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (20/48)" #2259 daemon prio=5 os_prio=0 tid=0x00007f3b94270000 nid=0xb087 waiting on condition [0x00007f3af4e57000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9e7cdf78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Async calls on phase 1 (18/48)" #2258 daemon prio=5 os_prio=0 tid=0x00007f3b9427a800 nid=0xb086 waiting on condition [0x00007f3af29ec000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d9fbd9b20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-11" #2125 prio=10 os_prio=0 tid=0x00007f3a22185000 nid=0xaff8 runnable [0x00007f3ae13d2000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c04de8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c04dd0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c04d58> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-10" #2124 prio=10 os_prio=0 tid=0x00007f3a22180000 nid=0xaff7 runnable [0x00007f3ae14d3000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94bf2658> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94bf2640> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94bf25c8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-404-usw1-cl1-ms-mdb11:27017" #2168 daemon prio=5 os_prio=0 tid=0x00007f3a2217e800 nid=0xaff6 waiting on condition [0x00007f3ae15d4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c02c28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c02c50> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-2" #2116 prio=10 os_prio=0 tid=0x00007f3a2217d800 nid=0xaff5 runnable [0x00007f3ae16d5000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c02e00> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c02de8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c02d70> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-5" #2119 prio=10 os_prio=0 tid=0x00007f3a2217c000 nid=0xaff4 runnable [0x00007f3ae17d6000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c04fe0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c04fc8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c04f50> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-3" #2117 prio=10 os_prio=0 tid=0x00007f3a2217a800 nid=0xaff3 runnable [0x00007f3ae18d7000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c02ff8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c02fe0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c02f68> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-1" #2115 prio=10 os_prio=0 tid=0x00007f3a22179000 nid=0xaff2 runnable [0x00007f3ae19d8000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c051d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c051c0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c05148> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-6" #2120 prio=10 os_prio=0 tid=0x00007f3a22175000 nid=0xaff1 runnable [0x00007f3ae1ad9000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94bf2a08> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94bf29f0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94bf2978> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-8" #2122 prio=10 os_prio=0 tid=0x00007f3a22174000 nid=0xaff0 runnable [0x00007f3ae1bda000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c031f0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c031d8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c03160> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-4" #2118 prio=10 os_prio=0 tid=0x00007f3a22173800 nid=0xafef runnable [0x00007f3ae1cdb000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c053d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c053b8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c05340> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-22" #2167 daemon prio=5 os_prio=0 tid=0x00007f3a22170000 nid=0xafee waiting on condition [0x00007f3ae1ddc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf2cb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-9" #2123 prio=10 os_prio=0 tid=0x00007f3a2216c800 nid=0xafed runnable [0x00007f3ae1edd000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c033e8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c033d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c03358> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-4-7" #2121 prio=10 os_prio=0 tid=0x00007f3a2216a800 nid=0xafec runnable [0x00007f3ae1fde000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d94c055c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d94c055b0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d94c05538> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-403-usw1-cl1-ms-mdb10:27017" #2166 daemon prio=5 os_prio=0 tid=0x00007f3a22168800 nid=0xafeb waiting on condition [0x00007f3ae20df000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c034e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03510> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-21" #2165 daemon prio=5 os_prio=0 tid=0x00007f3a22167800 nid=0xafea waiting on condition [0x00007f3ae21e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c056c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-402-usw1-cl1-ms-mdb9:27017" #2164 daemon prio=5 os_prio=0 tid=0x00007f3a22166800 nid=0xafe9 waiting on condition [0x00007f3ae22e1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf2fa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94bf2fc8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-20" #2163 daemon prio=5 os_prio=0 tid=0x00007f3a22164000 nid=0xafe8 waiting on condition [0x00007f3ae23e2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c035c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-401-usw1-cl1-ms-mdb8:27017" #2162 daemon prio=5 os_prio=0 tid=0x00007f3a22161000 nid=0xafe7 waiting on condition [0x00007f3ae24e3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03840> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03868> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-19" #2161 daemon prio=5 os_prio=0 tid=0x00007f3a2215f800 nid=0xafe6 waiting on condition [0x00007f3ae25e4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c05800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-400-usw1-cl1-ms-mdb7:27017" #2160 daemon prio=5 os_prio=0 tid=0x00007f3a1e8ad000 nid=0xafe5 waiting on condition [0x00007f3ae26e5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c92e78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c92ea0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-18" #2159 daemon prio=5 os_prio=0 tid=0x00007f3a1e8af000 nid=0xafe4 waiting on condition [0x00007f3ae27e6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c05938> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-289-usw1-cl1-ms-mdb2:27017" #2158 daemon prio=5 os_prio=0 tid=0x00007f3a1e8aa000 nid=0xafe3 waiting on condition [0x00007f3ae28e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c05a70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c05a98> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-17" #2157 daemon prio=5 os_prio=0 tid=0x00007f3a1e8a9000 nid=0xafe2 waiting on condition [0x00007f3ae29e8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf3230> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-22-usw1-cl1-ms-mdb3:27017" #2156 daemon prio=5 os_prio=0 tid=0x00007f3a1e8a8800 nid=0xafe1 waiting on condition [0x00007f3ae2ae9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03948> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-16" #2155 daemon prio=5 os_prio=0 tid=0x00007f3a1e8ac000 nid=0xafe0 waiting on condition [0x00007f3ae2bea000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb46e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-20-usw1-cl1-ms-mdb1:27017" #2154 daemon prio=5 os_prio=0 tid=0x00007f3a1e8ab800 nid=0xafdf waiting on condition [0x00007f3ae2ceb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c92f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c92f80> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-15" #2153 daemon prio=5 os_prio=0 tid=0x00007f3a1e83c000 nid=0xafde waiting on condition [0x00007f3ae2dec000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-185-usw1-cl1-ms-mdb6:27017" #2152 daemon prio=5 os_prio=0 tid=0x00007f3a1e83b800 nid=0xafdd waiting on condition [0x00007f3ae2eed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf3518> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94bf3540> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-14" #2151 daemon prio=5 os_prio=0 tid=0x00007f3a1e83a800 nid=0xafdc waiting on condition [0x00007f3ae2fee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c93038> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-184-usw1-cl1-ms-mdb5:27017" #2150 daemon prio=5 os_prio=0 tid=0x00007f3a1e83a000 nid=0xafdb waiting on condition [0x00007f3af9157000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03b60> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-13" #2149 daemon prio=5 os_prio=0 tid=0x00007f3a1e838800 nid=0xafda waiting on condition [0x00007f3af9258000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf93b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316282', description='null'}-prod-183-usw1-cl1-ms-mdb4:27017" #2148 daemon prio=5 os_prio=0 tid=0x00007f3a1e838000 nid=0xafd9 waiting on condition [0x00007f3af9359000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03c40> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-12" #2147 daemon prio=5 os_prio=0 tid=0x00007f3a1e837000 nid=0xafd8 waiting on condition [0x00007f3af945a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03cf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"pool-3-thread-1" #2114 daemon prio=5 os_prio=0 tid=0x00007f3a1e865800 nid=0xafd7 waiting on condition [0x00007f3af955b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf96a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-404-usw1-cl1-ms-mdb11:27017" #2113 daemon prio=5 os_prio=0 tid=0x00007f3a1e864000 nid=0xafd6 waiting on condition [0x00007f3af965c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03e30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03e58> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-11" #2112 daemon prio=5 os_prio=0 tid=0x00007f3a1e863000 nid=0xafd5 waiting on condition [0x00007f3af975d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf9988> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-403-usw1-cl1-ms-mdb10:27017" #2111 daemon prio=5 os_prio=0 tid=0x00007f3a1e862000 nid=0xafd4 waiting on condition [0x00007f3af985e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c03f38> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-10" #2110 daemon prio=5 os_prio=0 tid=0x00007f3a1e860000 nid=0xafd3 waiting on condition [0x00007f3af995f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf9c70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-402-usw1-cl1-ms-mdb9:27017" #2108 daemon prio=5 os_prio=0 tid=0x00007f3a1e83d800 nid=0xafd1 waiting on condition [0x00007f3af9b61000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c03ff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c04018> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-9" #2106 daemon prio=5 os_prio=0 tid=0x00007f3a1e831800 nid=0xafcf waiting on condition [0x00007f3af9d63000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bf9f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-401-usw1-cl1-ms-mdb8:27017" #2105 daemon prio=5 os_prio=0 tid=0x00007f3a1e831000 nid=0xafce waiting on condition [0x00007f3af9e64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c040d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c040f8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-8" #2104 daemon prio=5 os_prio=0 tid=0x00007f3a1e830000 nid=0xafcd waiting on condition [0x00007f3af9f65000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c041b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-400-usw1-cl1-ms-mdb7:27017" #2103 daemon prio=5 os_prio=0 tid=0x00007f3a1e82f800 nid=0xafcc waiting on condition [0x00007f3afa066000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfa240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94bfa268> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-7" #2102 daemon prio=5 os_prio=0 tid=0x00007f3a1e82e800 nid=0xafcb waiting on condition [0x00007f3afa167000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c042e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-289-usw1-cl1-ms-mdb2:27017" #2101 daemon prio=5 os_prio=0 tid=0x00007f3a1e82e000 nid=0xafca waiting on condition [0x00007f3afa268000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfca30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94bfca58> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-6" #2100 daemon prio=5 os_prio=0 tid=0x00007f3a1e82d000 nid=0xafc9 waiting on condition [0x00007f3afa369000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c04420> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-22-usw1-cl1-ms-mdb3:27017" #2099 daemon prio=5 os_prio=0 tid=0x00007f3a1e82c800 nid=0xafc8 waiting on condition [0x00007f3afa46a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfccc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94bfcce8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-5" #2098 daemon prio=5 os_prio=0 tid=0x00007f3a1e819800 nid=0xafc7 waiting on condition [0x00007f3afa56b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c04558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-20-usw1-cl1-ms-mdb1:27017" #2097 daemon prio=5 os_prio=0 tid=0x00007f3a1e82b000 nid=0xafc6 waiting on condition [0x00007f3afa66c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c04690> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c046b8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-4" #2096 daemon prio=5 os_prio=0 tid=0x00007f3a1e817800 nid=0xafc5 waiting on condition [0x00007f3afa76d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfcf50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-185-usw1-cl1-ms-mdb6:27017" #2095 daemon prio=5 os_prio=0 tid=0x00007f3a1e829000 nid=0xafc4 waiting on condition [0x00007f3afa86e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c04770> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c04798> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-3" #2094 daemon prio=5 os_prio=0 tid=0x00007f3a1e828000 nid=0xafc3 waiting on condition [0x00007f3b017b0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfd238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-184-usw1-cl1-ms-mdb5:27017" #2092 daemon prio=5 os_prio=0 tid=0x00007f3a1e826800 nid=0xafc1 waiting on condition [0x00007f3b019b2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c04850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c04878> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-2" #2091 daemon prio=5 os_prio=0 tid=0x00007f3a1e81c800 nid=0xafc0 waiting on condition [0x00007f3b01ab3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfd520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b294b1d65b0b983b316281', description='null'}-prod-183-usw1-cl1-ms-mdb4:27017" #2090 daemon prio=5 os_prio=0 tid=0x00007f3a1e826000 nid=0xafbf waiting on condition [0x00007f3b01bb4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c91c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c91c38> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" #2089 daemon prio=5 os_prio=0 tid=0x00007f3a1e821000 nid=0xafbe waiting on condition [0x00007f3b01cb5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94bfd808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (26/48)" #1842 daemon prio=5 os_prio=0 tid=0x00007f3a4b936800 nid=0xae98 waiting on condition [0x00007f3af5962000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb4b78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (22/48)" #1841 daemon prio=5 os_prio=0 tid=0x00007f3a4581d000 nid=0xae97 waiting on condition [0x00007f3af5a63000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c927c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (25/48)" #1840 daemon prio=5 os_prio=0 tid=0x00007f3a45810000 nid=0xae94 waiting on condition [0x00007f3af5b64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c05e30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (31/48)" #1839 daemon prio=5 os_prio=0 tid=0x00007f3a4ba45000 nid=0xae93 waiting on condition [0x00007f3afad73000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c05f68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (17/48)" #1837 daemon prio=5 os_prio=0 tid=0x00007f3a4580e000 nid=0xae90 waiting on condition [0x00007f3af8e70000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c07148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (18/48)" #1836 daemon prio=5 os_prio=0 tid=0x00007f3a37800800 nid=0xae8a waiting on condition [0x00007f3afb379000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c060a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 15" #71 daemon prio=5 os_prio=0 tid=0x00007f3a48800800 nid=0xae83 runnable [0x00007f3af6069000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182fcb10> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182fb8a0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182fb760> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (23/48)" #1835 daemon prio=5 os_prio=0 tid=0x00007f3a51859000 nid=0xae82 waiting on condition [0x00007f3afae74000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb4cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (29/48)" #1834 daemon prio=5 os_prio=0 tid=0x00007f3a6f062800 nid=0xae80 waiting on condition [0x00007f3afac72000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c07280> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (28/48)" #1832 daemon prio=5 os_prio=0 tid=0x00007f3a4b90d800 nid=0xae76 waiting on condition [0x00007f3af646d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c061d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (19/48)" #1831 daemon prio=5 os_prio=0 tid=0x00007f3a4b92c800 nid=0xae75 waiting on condition [0x00007f3af5c65000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb4de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (20/48)" #1807 daemon prio=5 os_prio=0 tid=0x00007f3a45809800 nid=0xae5b waiting on condition [0x00007f3af8f71000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c06310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (30/48)" #1795 daemon prio=5 os_prio=0 tid=0x00007f3a4b942000 nid=0xae4c waiting on condition [0x00007f3afbd83000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c073b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (32/48)" #1753 daemon prio=5 os_prio=0 tid=0x00007f3a4b900000 nid=0xae1f waiting on condition [0x00007f3b00bb0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb5520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (24/48)" #1752 daemon prio=5 os_prio=0 tid=0x00007f3a6301d800 nid=0xae19 waiting on condition [0x00007f3b00cb1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c06a48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (27/48)" #1751 daemon prio=5 os_prio=0 tid=0x00007f3a6f090000 nid=0xae17 waiting on condition [0x00007f3b00db2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb5658> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 15" #87 daemon prio=5 os_prio=0 tid=0x00007f3c0819b000 nid=0xadfa runnable [0x00007f3b00eb3000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1835dce0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1835ca70> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1835c930> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 14" #70 daemon prio=5 os_prio=0 tid=0x00007f3ae4964000 nid=0xadf7 runnable [0x00007f3b00fb4000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182f6fe0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182f5d70> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182f5c30> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 13" #69 daemon prio=5 os_prio=0 tid=0x00007f3ae4963000 nid=0xadf6 runnable [0x00007f3b010b5000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182f14b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182f0240> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182f0100> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Time Trigger for phase 4 (21/48)" #1749 daemon prio=5 os_prio=0 tid=0x00007f3ae4962000 nid=0xadf5 waiting on condition [0x00007f3b011b6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cb5790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-1" #1748 daemon prio=5 os_prio=0 tid=0x00007f3ae4958000 nid=0xadf4 waiting on condition [0x00007f3b014b7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c06b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b2947ad65b0b983b316280', description='null'}-prod-362-usw1-cl1-mgmt-mdb5:27017" #1747 daemon prio=5 os_prio=0 tid=0x00007f3ae4951800 nid=0xadf3 waiting on condition [0x00007f3b015b8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cd0458> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94cd0480> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b2947ad65b0b983b316280', description='null'}-prod-27-usw1-cl1-mgmt-mdb2:27017" #1746 daemon prio=5 os_prio=0 tid=0x00007f3ae4950800 nid=0xadf2 waiting on condition [0x00007f3b01fb6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94cd0538> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94cd0560> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b2947ad65b0b983b316280', description='null'}-prod-26-usw1-cl1-mgmt-mdb1:27017" #1745 daemon prio=5 os_prio=0 tid=0x00007f3ae494c800 nid=0xadf1 waiting on condition [0x00007f3b020b7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d94c06cb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d94c06ce0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 14" #86 daemon prio=5 os_prio=0 tid=0x00007f3c0814f800 nid=0xade0 runnable [0x00007f3b021b8000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d183581b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18356f40> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18356e00> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 13" #85 daemon prio=5 os_prio=0 tid=0x00007f3c08012000 nid=0xaddc runnable [0x00007f3b022b9000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18352680> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18351410> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d183512d0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 12" #84 daemon prio=5 os_prio=0 tid=0x00007f3c08151000 nid=0xadd5 runnable [0x00007f3b023ba000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1834cb50> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1834b8e0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1834b7a0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 12" #68 daemon prio=5 os_prio=0 tid=0x00007f3ae4910000 nid=0xadcf runnable [0x00007f3b092ad000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182eb980> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182ea710> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182ea5d0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 11" #67 daemon prio=5 os_prio=0 tid=0x00007f3ae69c7000 nid=0xadce runnable [0x00007f3b090ab000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182e5e50> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182e4be0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182e4aa0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 11" #83 daemon prio=5 os_prio=0 tid=0x00007f3c0800b000 nid=0xadba runnable [0x00007f3b091ac000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18347020> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18345db0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18345c70> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 10" #66 daemon prio=5 os_prio=0 tid=0x00007f3ba7f12000 nid=0xada7 runnable [0x00007f3b093ae000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182e0320> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182df0b0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182def70> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 9" #65 daemon prio=5 os_prio=0 tid=0x00007f3ba7ef1000 nid=0xada6 runnable [0x00007f3b094af000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182da7f0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182d9580> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182d9440> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 8" #64 daemon prio=5 os_prio=0 tid=0x00007f46f8001000 nid=0xada5 runnable [0x00007f3b095b0000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182d4cc0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182d3a50> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182d3910> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"BackpressureMonitor" #1699 daemon prio=5 os_prio=0 tid=0x00007f3c7cf2b000 nid=0xada4 waiting on condition [0x00007f3b096b1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at com.utils.BackpressureMonitor.run(BackpressureMonitor.java:78)

"CloseableReaperThread" #1697 daemon prio=5 os_prio=0 tid=0x00007f3ba763f000 nid=0xab99 in Object.wait() [0x00007f3b097b2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c8d760> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (32/48)" #1695 daemon prio=5 os_prio=0 tid=0x00007f3b94277000 nid=0xab98 in Object.wait() [0x00007f3b099b4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3af22a70> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1696 daemon prio=5 os_prio=0 tid=0x00007f3ba75bf800 nid=0xab97 in Object.wait() [0x00007f3b09cb9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c8d740> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1694 daemon prio=5 os_prio=0 tid=0x00007f3ba75a3800 nid=0xab96 in Object.wait() [0x00007f3b09dba000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c8d770> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1693 daemon prio=5 os_prio=0 tid=0x00007f3ba642a800 nid=0xab95 in Object.wait() [0x00007f3b09ebb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c8d750> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (31/48)" #1692 daemon prio=5 os_prio=0 tid=0x00007f3b94276000 nid=0xab94 in Object.wait() [0x00007f3b0a1bc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3af6ce40> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"Sink: phase 5 (30/48)" #1691 daemon prio=5 os_prio=0 tid=0x00007f3b9426f000 nid=0xab93 in Object.wait() [0x00007f3b0a2bd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3aefe6e0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"Sink: phase 5 (29/48)" #1690 daemon prio=5 os_prio=0 tid=0x00007f3b9426e000 nid=0xab92 in Object.wait() [0x00007f3b0a3be000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3ae44d20> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1689 daemon prio=5 os_prio=0 tid=0x00007f3ba5f26800 nid=0xab91 in Object.wait() [0x00007f3b0a4bf000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c29fd0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (28/48)" #1688 daemon prio=5 os_prio=0 tid=0x00007f3b9426c800 nid=0xab90 in Object.wait() [0x00007f3b0a5c0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a906360> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1687 daemon prio=5 os_prio=0 tid=0x00007f3ba5f15800 nid=0xab8f in Object.wait() [0x00007f3b0a6c1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2ab08> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (27/48)" #1686 daemon prio=5 os_prio=0 tid=0x00007f3b9426b800 nid=0xab8e in Object.wait() [0x00007f3b0a7c2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3af466f8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1685 daemon prio=5 os_prio=0 tid=0x00007f3ba5f1c000 nid=0xab8d in Object.wait() [0x00007f3b0a8c3000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2b450> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1684 daemon prio=5 os_prio=0 tid=0x00007f3ba5f0b000 nid=0xab8c in Object.wait() [0x00007f3b0a9c4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2b678> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1683 daemon prio=5 os_prio=0 tid=0x00007f3ba5f0e000 nid=0xab8b in Object.wait() [0x00007f3b0aac5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2b8a0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (26/48)" #1682 daemon prio=5 os_prio=0 tid=0x00007f3b9426a800 nid=0xab8a in Object.wait() [0x00007f3b0abc6000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a8e3970> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"Sink: phase 5 (25/48)" #1681 daemon prio=5 os_prio=0 tid=0x00007f3b9427d800 nid=0xab89 in Object.wait() [0x00007f3b0acc7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a8bd9d0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1680 daemon prio=5 os_prio=0 tid=0x00007f3ba5f13800 nid=0xab88 in Object.wait() [0x00007f3b0adc8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2c938> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (24/48)" #1679 daemon prio=5 os_prio=0 tid=0x00007f3b9427c800 nid=0xab87 in Object.wait() [0x00007f3b0b3d5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3ae830a0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"Sink: phase 5 (23/48)" #1677 daemon prio=5 os_prio=0 tid=0x00007f3b9427b800 nid=0xab86 in Object.wait() [0x00007f3b0b1d3000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a8a8b68> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1678 daemon prio=5 os_prio=0 tid=0x00007f3ba5f12800 nid=0xab85 in Object.wait() [0x00007f3b0b2d4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2d9c0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (22/48)" #1675 daemon prio=5 os_prio=0 tid=0x00007f3b94217800 nid=0xab83 in Object.wait() [0x00007f3b0b4d6000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a891688> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1674 daemon prio=5 os_prio=0 tid=0x00007f3ba5f10800 nid=0xab82 in Object.wait() [0x00007f3b0b5d7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2e5a0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (21/48)" #1673 daemon prio=5 os_prio=0 tid=0x00007f3b94216800 nid=0xab81 in Object.wait() [0x00007f3b0b6d8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a87ee58> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1672 daemon prio=5 os_prio=0 tid=0x00007f3ba5f0c000 nid=0xab80 in Object.wait() [0x00007f3b0b9d4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2eee8> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1671 daemon prio=5 os_prio=0 tid=0x00007f3ba5f09000 nid=0xab7f in Object.wait() [0x00007f3b0bad5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c2f110> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"Sink: phase 5 (20/48)" #1670 daemon prio=5 os_prio=0 tid=0x00007f3b94215000 nid=0xab7e in Object.wait() [0x00007f3b0bbd6000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a86e938> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"Sink: phase 5 (19/48)" #1667 daemon prio=5 os_prio=0 tid=0x00007f3b94214000 nid=0xab7d in Object.wait() [0x00007f3b0bcd7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a81f960> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1669 daemon prio=5 os_prio=0 tid=0x00007f3ba5f06800 nid=0xab7c in Object.wait() [0x00007f3b0bdd8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c302f0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1668 daemon prio=5 os_prio=0 tid=0x00007f3ba5f06000 nid=0xab7b in Object.wait() [0x00007f3b0bed9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c30608> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1666 daemon prio=5 os_prio=0 tid=0x00007f3ba5ee7000 nid=0xab7a waiting on condition [0x00007f3b0bfda000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"Sink: phase 5 (18/48)" #1665 daemon prio=5 os_prio=0 tid=0x00007f3b9425c000 nid=0xab78 in Object.wait() [0x00007f3b0b7d9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a8007b8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1664 daemon prio=5 os_prio=0 tid=0x00007f3ba5eeb000 nid=0xab71 waiting on condition [0x00007f3b0c3de000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"Sink: phase 5 (17/48)" #1662 daemon prio=5 os_prio=0 tid=0x00007f3b9425b000 nid=0xab6f in Object.wait() [0x00007f3b0afd8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d3a7f6088> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1663 daemon prio=5 os_prio=0 tid=0x00007f3ba5eee000 nid=0xab69 in Object.wait() [0x00007f3b0c1dc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c31a28> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1661 daemon prio=5 os_prio=0 tid=0x00007f3ba5eec000 nid=0xab64 in Object.wait() [0x00007f3b0c2dd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c31c50> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (32/48)" #1656 daemon prio=5 os_prio=0 tid=0x00007f3b9425a000 nid=0xab47 in Object.wait() [0x00007f3b0c6e1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d53ac7f60> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1658 daemon prio=5 os_prio=0 tid=0x00007f3ba5ef3000 nid=0xab45 waiting on condition [0x00007f3b0c8e3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1657 daemon prio=5 os_prio=0 tid=0x00007f3bc52e8000 nid=0xab3c waiting on condition [0x00007f3b0c9e4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1655 daemon prio=5 os_prio=0 tid=0x00007f3ba5ee4800 nid=0xab39 waiting on condition [0x00007f3b0cae5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1654 daemon prio=5 os_prio=0 tid=0x00007f3ba5ed7800 nid=0xab35 waiting on condition [0x00007f3b0cbe6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 4 (31/48)" #1653 daemon prio=5 os_prio=0 tid=0x00007f3b94259000 nid=0xab32 in Object.wait() [0x00007f3b0cce7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d53a97ab8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1652 daemon prio=5 os_prio=0 tid=0x00007f3ba5ed6800 nid=0xab2c waiting on condition [0x00007f3b0cde8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1651 daemon prio=5 os_prio=0 tid=0x00007f3ba5ee0000 nid=0xab2b in Object.wait() [0x00007f3b0cee9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c33760> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (30/48)" #1649 daemon prio=5 os_prio=0 tid=0x00007f3b94289800 nid=0xab2a in Object.wait() [0x00007f3b0f6b7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d53a7c490> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1650 daemon prio=5 os_prio=0 tid=0x00007f3ba5edf000 nid=0xab29 waiting on condition [0x00007f3b0f7b8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1648 daemon prio=5 os_prio=0 tid=0x00007f3ba5ed5000 nid=0xab28 in Object.wait() [0x00007f3b0f8b9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c34320> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1647 daemon prio=5 os_prio=0 tid=0x00007f3ba5d92800 nid=0xab27 in Object.wait() [0x00007f3b0f9ba000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c34548> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (29/48)" #1646 daemon prio=5 os_prio=0 tid=0x00007f3b94288800 nid=0xab26 in Object.wait() [0x00007f3b0fabb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d539c7e38> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 4 (28/48)" #1645 daemon prio=5 os_prio=0 tid=0x00007f3b94287000 nid=0xab25 in Object.wait() [0x00007f3b0fbbc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d538a1a18> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1644 daemon prio=5 os_prio=0 tid=0x00007f3ba5d94800 nid=0xab24 in Object.wait() [0x00007f3b0fcbd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c37b68> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1643 daemon prio=5 os_prio=0 tid=0x00007f3ba5eda800 nid=0xab23 in Object.wait() [0x00007f3b104ed000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c37d90> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (27/48)" #1642 daemon prio=5 os_prio=0 tid=0x00007f3b94284800 nid=0xab22 in Object.wait() [0x00007f3b105ee000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d5385efb8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1641 daemon prio=5 os_prio=0 tid=0x00007f3bc52bd800 nid=0xab21 in Object.wait() [0x00007f3b10af3000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c38ac8> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (26/48)" #1640 daemon prio=5 os_prio=0 tid=0x00007f3b94283800 nid=0xab20 in Object.wait() [0x00007f3b109f2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d539e7718> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1639 daemon prio=5 os_prio=0 tid=0x00007f3bc52d3000 nid=0xab1f waiting on condition [0x00007f3b106ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 4 (25/48)" #1634 daemon prio=5 os_prio=0 tid=0x00007f3b94282800 nid=0xab1e in Object.wait() [0x00007f3b107f0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d5382f708> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1638 daemon prio=5 os_prio=0 tid=0x00007f3bc52c0800 nid=0xab1d waiting on condition [0x00007f3b108f1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1635 daemon prio=5 os_prio=0 tid=0x00007f3bc52af800 nid=0xab18 waiting on condition [0x00007f3b110eb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1633 daemon prio=5 os_prio=0 tid=0x00007f3bc52b0800 nid=0xab13 in Object.wait() [0x00007f3b111ec000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3a158> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (24/48)" #1632 daemon prio=5 os_prio=0 tid=0x00007f3b94281000 nid=0xab10 in Object.wait() [0x00007f3b112ed000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d5380ab78> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1631 daemon prio=5 os_prio=0 tid=0x00007f3bc52bc800 nid=0xab07 in Object.wait() [0x00007f3b113ee000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3ab20> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (23/48)" #1630 daemon prio=5 os_prio=0 tid=0x00007f3b94212800 nid=0xab06 in Object.wait() [0x00007f3b119ed000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d538e1f90> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1629 daemon prio=5 os_prio=0 tid=0x00007f3bc52b7800 nid=0xab05 in Object.wait() [0x00007f3b11aee000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3b498> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (22/48)" #1628 daemon prio=5 os_prio=0 tid=0x00007f3b94211000 nid=0xab04 in Object.wait() [0x00007f3b11bef000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d537eed68> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1627 daemon prio=5 os_prio=0 tid=0x00007f3ba5edd000 nid=0xab03 waiting on condition [0x00007f3b11cf0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1626 daemon prio=5 os_prio=0 tid=0x00007f3bc52b6000 nid=0xab02 waiting on condition [0x00007f3b121e7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1625 daemon prio=5 os_prio=0 tid=0x00007f3ba5edc000 nid=0xab01 waiting on condition [0x00007f3b122e8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1624 daemon prio=5 os_prio=0 tid=0x00007f3bc5296000 nid=0xab00 waiting on condition [0x00007f3b123e9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1623 daemon prio=5 os_prio=0 tid=0x00007f3ba5d96000 nid=0xaaff waiting on condition [0x00007f3b124ea000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1622 daemon prio=5 os_prio=0 tid=0x00007f3ba5d95800 nid=0xaafe in Object.wait() [0x00007f3b125eb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3c718> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1621 daemon prio=5 os_prio=0 tid=0x00007f3bc4e11800 nid=0xaafd in Object.wait() [0x00007f3b126ec000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3c940> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1620 daemon prio=5 os_prio=0 tid=0x00007f3ba5d8e800 nid=0xaafc in Object.wait() [0x00007f3b127ed000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3cb68> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (21/48)" #1619 daemon prio=5 os_prio=0 tid=0x00007f3b94210000 nid=0xaafb in Object.wait() [0x00007f3b128ee000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d537c6e00> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 4 (20/48)" #1618 daemon prio=5 os_prio=0 tid=0x00007f3b942d0000 nid=0xaaf8 in Object.wait() [0x00007f3b11df1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d537a4648> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 4 (19/48)" #1615 daemon prio=5 os_prio=0 tid=0x00007f3b942cf000 nid=0xaaf4 in Object.wait() [0x00007f3b11ef2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d53775830> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1617 daemon prio=5 os_prio=0 tid=0x00007f3bc5169000 nid=0xaaef in Object.wait() [0x00007f3b11ff3000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3e938> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1616 daemon prio=5 os_prio=0 tid=0x00007f3ba5d89000 nid=0xaae0 waiting on condition [0x00007f3b114ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1614 daemon prio=5 os_prio=0 tid=0x00007f3ba5d88000 nid=0xaadb in Object.wait() [0x00007f3b115f0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3ed28> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1613 daemon prio=5 os_prio=0 tid=0x00007f3bc514c800 nid=0xaad8 waiting on condition [0x00007f3b116f1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1612 daemon prio=5 os_prio=0 tid=0x00007f3ba5d87000 nid=0xaad3 in Object.wait() [0x00007f3b117f2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c3f118> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 4 (18/48)" #1611 daemon prio=5 os_prio=0 tid=0x00007f3b942ce000 nid=0xaacf in Object.wait() [0x00007f3b129ef000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d5375eec8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 4 (17/48)" #1610 daemon prio=5 os_prio=0 tid=0x00007f3b942cd000 nid=0xaaca in Object.wait() [0x00007f3b12af0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d5374ae48> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 3 (32/48)" #1608 daemon prio=5 os_prio=0 tid=0x00007f3b94009800 nid=0xaac6 in Object.wait() [0x00007f3b12bf1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c406c8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1605 daemon prio=5 os_prio=0 tid=0x00007f3bc4e25800 nid=0xaabe in Object.wait() [0x00007f3b12cf2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c40e40> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1609 daemon prio=5 os_prio=0 tid=0x00007f3bc4d9f800 nid=0xaabb waiting on condition [0x00007f3b12df3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1607 daemon prio=5 os_prio=0 tid=0x00007f3bc4b23000 nid=0xaab9 waiting on condition [0x00007f3b12ef4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1606 daemon prio=5 os_prio=0 tid=0x00007f3bc4e19800 nid=0xaab4 waiting on condition [0x00007f3b12ff5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1604 daemon prio=5 os_prio=0 tid=0x00007f3ba54c8000 nid=0xaab2 waiting on condition [0x00007f3b137f6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1603 daemon prio=5 os_prio=0 tid=0x00007f3bc4e1b800 nid=0xaaae in Object.wait() [0x00007f3b138f7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c41930> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (31/48)" #1602 daemon prio=5 os_prio=0 tid=0x00007f3b942d3800 nid=0xaaa2 in Object.wait() [0x00007f3b139f8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c41b48> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1601 daemon prio=5 os_prio=0 tid=0x00007f3bc4e1c800 nid=0xaa94 in Object.wait() [0x00007f3b13af9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c422c0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (30/48)" #1598 daemon prio=5 os_prio=0 tid=0x00007f3b942cb800 nid=0xaa8d in Object.wait() [0x00007f3b13bfa000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c424d8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1600 daemon prio=5 os_prio=0 tid=0x00007f3bc4e24000 nid=0xaa8a in Object.wait() [0x00007f3b13cfb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c42c70> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1599 daemon prio=5 os_prio=0 tid=0x00007f3ba5b1f800 nid=0xaa81 in Object.wait() [0x00007f3b13dfc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c42e98> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1597 daemon prio=5 os_prio=0 tid=0x00007f3ba5b05000 nid=0xaa78 waiting on condition [0x00007f3b0ffef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 3 (29/48)" #1596 daemon prio=5 os_prio=0 tid=0x00007f3b942ca800 nid=0xaa5f in Object.wait() [0x00007f3b100f0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c43278> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1595 daemon prio=5 os_prio=0 tid=0x00007f3ba57f2000 nid=0xaa55 in Object.wait() [0x00007f3b101f1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c439f0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (28/48)" #1594 daemon prio=5 os_prio=0 tid=0x00007f3b94003000 nid=0xa9eb in Object.wait() [0x00007f3b102f2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c43c20> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 3 (27/48)" #1593 daemon prio=5 os_prio=0 tid=0x00007f3b94285800 nid=0xa9e4 in Object.wait() [0x00007f3b130f6000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c44388> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1592 daemon prio=5 os_prio=0 tid=0x00007f3bc4a01800 nid=0xa9df waiting on condition [0x00007f3b131f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 3 (26/48)" #1591 daemon prio=5 os_prio=0 tid=0x00007f3b94227000 nid=0xa9ca in Object.wait() [0x00007f3b132f8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c44cb8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1590 daemon prio=5 os_prio=0 tid=0x00007f3bc4c53800 nid=0xa9b5 in Object.wait() [0x00007f3b133f9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c45488> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1589 daemon prio=5 os_prio=0 tid=0x00007f3ba5801000 nid=0xa9a6 waiting on condition [0x00007f3b134fa000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 3 (25/48)" #1586 daemon prio=5 os_prio=0 tid=0x00007f3b9400b800 nid=0xa992 in Object.wait() [0x00007f3b135fb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d6c8dac60> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1588 daemon prio=5 os_prio=0 tid=0x00007f3bc4b31800 nid=0xa990 waiting on condition [0x00007f3b13efd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1587 daemon prio=5 os_prio=0 tid=0x00007f3ba56ed800 nid=0xa986 in Object.wait() [0x00007f3b10bf4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c46290> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1585 daemon prio=5 os_prio=0 tid=0x00007f3ba55cd800 nid=0xa970 waiting on condition [0x00007f3b10cf5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1584 daemon prio=5 os_prio=0 tid=0x00007f3ba55d2800 nid=0xa968 waiting on condition [0x00007f3b10df6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1583 daemon prio=5 os_prio=0 tid=0x00007f3ba55d7800 nid=0xa95f waiting on condition [0x00007f3b10ef7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1582 daemon prio=5 os_prio=0 tid=0x00007f3bc49f5800 nid=0xa956 in Object.wait() [0x00007f3b50ac9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c46a10> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (24/48)" #1581 daemon prio=5 os_prio=0 tid=0x00007f3b94001000 nid=0xa951 in Object.wait() [0x00007f3b518d7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c46c28> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 3 (23/48)" #1580 daemon prio=5 os_prio=0 tid=0x00007f3b9422b800 nid=0xa950 in Object.wait() [0x00007f3b519d8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c474d8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1579 daemon prio=5 os_prio=0 tid=0x00007f3bc42e5800 nid=0xa94f waiting on condition [0x00007f3b51ad9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1578 daemon prio=5 os_prio=0 tid=0x00007f3ba54a7800 nid=0xa94e in Object.wait() [0x00007f3b51bda000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c47e18> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1577 daemon prio=5 os_prio=0 tid=0x00007f3ba54a6800 nid=0xa94d in Object.wait() [0x00007f3b51cdb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c48040> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"CloseableReaperThread" #1576 daemon prio=5 os_prio=0 tid=0x00007f3ba54a6000 nid=0xa94c in Object.wait() [0x00007f3b51ddc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c48268> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (22/48)" #1574 daemon prio=5 os_prio=0 tid=0x00007f3b942af000 nid=0xa94b in Object.wait() [0x00007f3b51edd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c48480> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1575 daemon prio=5 os_prio=0 tid=0x00007f3ba54a3000 nid=0xa949 waiting on condition [0x00007f3b513d9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 3 (21/48)" #1573 daemon prio=5 os_prio=0 tid=0x00007f3b942d5800 nid=0xa948 in Object.wait() [0x00007f3b514da000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d6c867b90> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1572 daemon prio=5 os_prio=0 tid=0x00007f3bc42e3800 nid=0xa942 in Object.wait() [0x00007f3b515db000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c49630> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 3 (20/48)" #1571 daemon prio=5 os_prio=0 tid=0x00007f3b942a3800 nid=0xa940 in Object.wait() [0x00007f3b516dc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c49988> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1570 daemon prio=5 os_prio=0 tid=0x00007f3ba54a0000 nid=0xa937 in Object.wait() [0x00007f3b51fde000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4a100> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1569 daemon prio=5 os_prio=0 tid=0x00007f3bc42c0800 nid=0xa936 waiting on condition [0x00007f3b520df000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 3 (19/48)" #1568 daemon prio=5 os_prio=0 tid=0x00007f3b942e1000 nid=0xa935 in Object.wait() [0x00007f3b521e0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4a4e0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"phase 3 (18/48)" #1567 daemon prio=5 os_prio=0 tid=0x00007f3b942d8000 nid=0xa931 in Object.wait() [0x00007f3b50edb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4ac98> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1566 daemon prio=5 os_prio=0 tid=0x00007f3bc423a000 nid=0xa921 in Object.wait() [0x00007f3b50fdc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4b410> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1565 daemon prio=5 os_prio=0 tid=0x00007f3ba535f000 nid=0xa91b waiting for monitor entry [0x00007f3b510dd000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d9a8b6ea8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"phase 3 (17/48)" #1564 daemon prio=5 os_prio=0 tid=0x00007f3b942da000 nid=0xa912 in Object.wait() [0x00007f3b511de000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4b7f0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1563 daemon prio=5 os_prio=0 tid=0x00007f3bc4238800 nid=0xa90f waiting for monitor entry [0x00007f3b522e1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d9a952158> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"CloseableReaperThread" #1562 daemon prio=5 os_prio=0 tid=0x00007f3ba5352000 nid=0xa90e in Object.wait() [0x00007f3b523e2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4c198> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (32/48)" #1561 daemon prio=5 os_prio=0 tid=0x00007f3b942dc800 nid=0xa90d in Object.wait() [0x00007f3b524e3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d6c5b0f70> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d9a8b6ea8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c4c5b8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1560 daemon prio=5 os_prio=0 tid=0x00007f3bc4236800 nid=0xa90c waiting on condition [0x00007f3b525e4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1559 daemon prio=5 os_prio=0 tid=0x00007f3bc4235800 nid=0xa90b in Object.wait() [0x00007f3b526e5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4ccf0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1558 daemon prio=5 os_prio=0 tid=0x00007f3bc4234800 nid=0xa90a sleeping[0x00007f3b527e6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 2 (31/48)" #1556 daemon prio=5 os_prio=0 tid=0x00007f3b942de800 nid=0xa909 in Object.wait() [0x00007f3b528e7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d6c6f7ae8> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d9a952158> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c4d460> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1557 daemon prio=5 os_prio=0 tid=0x00007f3ba5007800 nid=0xa908 sleeping[0x00007f3b529e8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1555 daemon prio=5 os_prio=0 tid=0x00007f3bc422a800 nid=0xa907 in Object.wait() [0x00007f3b52ae9000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4db98> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (30/48)" #1553 daemon prio=5 os_prio=0 tid=0x00007f3b942aa800 nid=0xa906 in Object.wait() [0x00007f3b52bea000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4ddb0> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1554 daemon prio=5 os_prio=0 tid=0x00007f3bc4229000 nid=0xa905 in Object.wait() [0x00007f3b52ceb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4e608> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (29/48)" #1552 daemon prio=5 os_prio=0 tid=0x00007f3b942b1000 nid=0xa904 in Object.wait() [0x00007f3b52dec000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4e820> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1551 daemon prio=5 os_prio=0 tid=0x00007f3ba5005800 nid=0xa903 in Object.wait() [0x00007f3b52eed000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4ef98> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1550 daemon prio=5 os_prio=0 tid=0x00007f3ba5004800 nid=0xa902 waiting on condition [0x00007f3b52fee000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 2 (28/48)" #1549 daemon prio=5 os_prio=0 tid=0x00007f3b9429d000 nid=0xa901 in Object.wait() [0x00007f3b530ef000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4f390> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1548 daemon prio=5 os_prio=0 tid=0x00007f3ba4fd9800 nid=0xa900 in Object.wait() [0x00007f3b531f0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c4fb08> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (27/48)" #1547 daemon prio=5 os_prio=0 tid=0x00007f3b94294000 nid=0xa8ff in Object.wait() [0x00007f3b532f1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c4fd20> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1546 daemon prio=5 os_prio=0 tid=0x00007f3ba4fd7000 nid=0xa8fe waiting for monitor entry [0x00007f3b533f2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d9acbb298> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1545 daemon prio=5 os_prio=0 tid=0x00007f3bc42e8800 nid=0xa8fd sleeping[0x00007f3b534f3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1544 daemon prio=5 os_prio=0 tid=0x00007f3ba4ed0000 nid=0xa8fc in Object.wait() [0x00007f3b535f4000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c50970> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (26/48)" #1543 daemon prio=5 os_prio=0 tid=0x00007f3b942a6000 nid=0xa8fb in Object.wait() [0x00007f3b536f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d6c7880f8> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d9acbb298> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c50d90> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1542 daemon prio=5 os_prio=0 tid=0x00007f3ba4dce800 nid=0xa8fa waiting for monitor entry [0x00007f3b537f6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d9ad5c160> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1541 daemon prio=5 os_prio=0 tid=0x00007f3bc424c800 nid=0xa8f9 waiting on condition [0x00007f3b538f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1540 daemon prio=5 os_prio=0 tid=0x00007f3bc4291000 nid=0xa8f8 in Object.wait() [0x00007f3b539f8000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c51690> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1539 daemon prio=5 os_prio=0 tid=0x00007f3ba4dba000 nid=0xa8f7 waiting on condition [0x00007f3b53af9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 2 (25/48)" #1537 daemon prio=5 os_prio=0 tid=0x00007f3b9429a800 nid=0xa8f6 in Object.wait() [0x00007f3b53bfa000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c51bb8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1538 daemon prio=5 os_prio=0 tid=0x00007f3bc4001800 nid=0xa8f5 in Object.wait() [0x00007f3b53cfb000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c52330> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (24/48)" #1534 daemon prio=5 os_prio=0 tid=0x00007f3b9429f000 nid=0xa8f4 in Object.wait() [0x00007f3b53dfc000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c52548> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1536 daemon prio=5 os_prio=0 tid=0x00007f3ba4da1000 nid=0xa8f3 in Object.wait() [0x00007f3b53efd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c52d10> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1535 daemon prio=5 os_prio=0 tid=0x00007f3bc4231800 nid=0xa8f2 sleeping[0x00007f3b53ffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"phase 2 (23/48)" #1533 daemon prio=5 os_prio=0 tid=0x00007f3b942a1800 nid=0xa8f1 in Object.wait() [0x00007f3c241cc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d6c853af0> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d9ad5c160> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c532f8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1532 daemon prio=5 os_prio=0 tid=0x00007f3ba416c800 nid=0xa8f0 in Object.wait() [0x00007f3c242cd000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c53868> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (22/48)" #1531 daemon prio=5 os_prio=0 tid=0x00007f3b942b3800 nid=0xa8ef in Object.wait() [0x00007f3c243ce000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c53a80> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1530 daemon prio=5 os_prio=0 tid=0x00007f3ba410a800 nid=0xa8ee waiting for monitor entry [0x00007f3c244cf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d9af8dd58> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"CloseableReaperThread" #1529 daemon prio=5 os_prio=0 tid=0x00007f3bc426f000 nid=0xa8ed in Object.wait() [0x00007f3c245d0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c54418> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (21/48)" #1528 daemon prio=5 os_prio=0 tid=0x00007f3b94298800 nid=0xa8ec in Object.wait() [0x00007f3c246d1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c54630> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1527 daemon prio=5 os_prio=0 tid=0x00007f3ba410e000 nid=0xa8eb sleeping[0x00007f3c247d2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1526 daemon prio=5 os_prio=0 tid=0x00007f3ba410c800 nid=0xa8ea in Object.wait() [0x00007f3c24ad3000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c55108> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (20/48)" #1525 daemon prio=5 os_prio=0 tid=0x00007f3b942b8000 nid=0xa8e9 in Object.wait() [0x00007f3c24bd4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d6c82e708> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d9af8dd58> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c55528> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1524 daemon prio=5 os_prio=0 tid=0x00007f3ba4151800 nid=0xa8e8 sleeping[0x00007f3c24cd5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"OutputFlusher" #1523 daemon prio=5 os_prio=0 tid=0x00007f3ba4150800 nid=0xa8e7 sleeping[0x00007f3c24dd6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:173)

"CloseableReaperThread" #1522 daemon prio=5 os_prio=0 tid=0x00007f3ba417e800 nid=0xa8e6 in Object.wait() [0x00007f3c25014000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c55e28> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (19/48)" #1521 daemon prio=5 os_prio=0 tid=0x00007f3b942b5800 nid=0xa8e5 in Object.wait() [0x00007f3c25115000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c56040> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1520 daemon prio=5 os_prio=0 tid=0x00007f3ba417f800 nid=0xa8e4 in Object.wait() [0x00007f3c25216000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c56800> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (18/48)" #1519 daemon prio=5 os_prio=0 tid=0x00007f3b942a8000 nid=0xa8e3 in Object.wait() [0x00007f3c25317000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c56a18> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1518 daemon prio=5 os_prio=0 tid=0x00007f3ba417d800 nid=0xa8e2 in Object.wait() [0x00007f3c25418000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c57190> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 2 (17/48)" #1517 daemon prio=5 os_prio=0 tid=0x00007f3b9422d000 nid=0xa8e1 in Object.wait() [0x00007f3c25519000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:452)
              - locked <0x00007f3d94c573a8> (a java.util.ArrayDeque)
              at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:153)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:192)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1516 daemon prio=5 os_prio=0 tid=0x00007f3ba4178800 nid=0xa8e0 waiting for monitor entry [0x00007f3c2561a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d3abfb120> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1515 daemon prio=5 os_prio=0 tid=0x00007f3ba4177800 nid=0xa8df waiting for monitor entry [0x00007f3c2571b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d60655618> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"CloseableReaperThread" #1514 daemon prio=5 os_prio=0 tid=0x00007f3ba4176800 nid=0xa8de in Object.wait() [0x00007f3c2581c000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c57ee0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (48/48)" #1513 daemon prio=5 os_prio=0 tid=0x00007f3c18008000 nid=0xa8dd in Object.wait() [0x00007f3c2591d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d3abbfa28> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d3abfb120> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c58300> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1512 daemon prio=5 os_prio=0 tid=0x00007f3ba4174000 nid=0xa8dc in Object.wait() [0x00007f3c25a1e000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c588c8> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (46/48)" #1511 daemon prio=5 os_prio=0 tid=0x00007f3c18007800 nid=0xa8db in Object.wait() [0x00007f3c25b1f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d606502c0> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d60655618> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c58ce8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"OutputFlusher" #1510 daemon prio=5 os_prio=0 tid=0x00007f3ba411b000 nid=0xa8da waiting for monitor entry [0x00007f3c25c20000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d94dd2a80> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1509 daemon prio=5 os_prio=0 tid=0x00007f3ba411a000 nid=0xa8d9 waiting for monitor entry [0x00007f3c263ec000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5c68f788> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1508 daemon prio=5 os_prio=0 tid=0x00007f3b24006800 nid=0xa8d8 waiting for monitor entry [0x00007f3c264ed000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5cbe2a98> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1507 daemon prio=5 os_prio=0 tid=0x00007f46f8129000 nid=0xa8d7 waiting for monitor entry [0x00007f3aaf46b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5cda77e0> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1506 daemon prio=5 os_prio=0 tid=0x00007f3ba4122000 nid=0xa8d6 waiting for monitor entry [0x00007f3b50dcc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5c45abe8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1505 daemon prio=5 os_prio=0 tid=0x00007f3bc8008800 nid=0xa8d5 waiting for monitor entry [0x00007f3aaf168000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5c1ef898> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1504 daemon prio=5 os_prio=0 tid=0x00007f3ba4120800 nid=0xa8d4 waiting for monitor entry [0x00007f3a89d22000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d66a905c8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1503 daemon prio=5 os_prio=0 tid=0x00007f3ba4186000 nid=0xa8d3 waiting for monitor entry [0x00007f3b50bca000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d53af3d70> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"CloseableReaperThread" #1501 daemon prio=5 os_prio=0 tid=0x00007f3ba4185800 nid=0xa8d2 in Object.wait() [0x00007f3b0c7e2000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5a188> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"OutputFlusher" #1502 daemon prio=5 os_prio=0 tid=0x00007f3b28151800 nid=0xa8d1 waiting for monitor entry [0x00007f3b0d0eb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d94d13078> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1500 daemon prio=5 os_prio=0 tid=0x00007f3c7c026000 nid=0xa8d0 waiting for monitor entry [0x00007f3aaf36a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5307f990> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1499 daemon prio=5 os_prio=0 tid=0x00007f3bd0174800 nid=0xa8cf waiting for monitor entry [0x00007f3b50ccb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d5c03ee48> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1498 daemon prio=5 os_prio=0 tid=0x00007f3bcc001000 nid=0xa8ce waiting for monitor entry [0x00007f3b0aec9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d53b27098> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1497 daemon prio=5 os_prio=0 tid=0x00007f473400d000 nid=0xa8cd waiting for monitor entry [0x00007f3b0cfea000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d663d7c80> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"OutputFlusher" #1496 daemon prio=5 os_prio=0 tid=0x00007f3ba417a800 nid=0xa8cc waiting for monitor entry [0x00007f3aaf269000]
   java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.flush(RecordWriter.java:175)
              - waiting to lock <0x00007f3d6027e030> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter$OutputFlusher.run(StreamRecordWriter.java:185)

"phase 1 (44/48)" #1495 daemon prio=5 os_prio=0 tid=0x00007f3c6074d800 nid=0xa8cb in Object.wait() [0x00007f3b505c4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d94dbbe30> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d94dd2a80> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5b080> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1494 daemon prio=5 os_prio=0 tid=0x00007f3ba418b800 nid=0xa8ca in Object.wait() [0x00007f3b507c6000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5b5f0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (42/48)" #1493 daemon prio=5 os_prio=0 tid=0x00007f3c6074d000 nid=0xa8c9 in Object.wait() [0x00007f3aaef65000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5c452a68> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5c45abe8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5ba10> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1492 daemon prio=5 os_prio=0 tid=0x00007f3ba4123800 nid=0xa8c8 in Object.wait() [0x00007f3b508c7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5bfd0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (40/48)" #1491 daemon prio=5 os_prio=0 tid=0x00007f3c6009e000 nid=0xa8c7 in Object.wait() [0x00007f3b509c8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d60277050> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d6027e030> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5c3f0> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1490 daemon prio=5 os_prio=0 tid=0x00007f3ba4003800 nid=0xa8c6 in Object.wait() [0x00007f3a95b38000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5c960> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (38/48)" #1489 daemon prio=5 os_prio=0 tid=0x00007f3c6009d000 nid=0xa8c5 in Object.wait() [0x00007f3ab5578000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d66a69840> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d66a905c8> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5cd80> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1488 daemon prio=5 os_prio=0 tid=0x00007f3ba4158800 nid=0xa8c4 in Object.wait() [0x00007f3ab146f000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5d338> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (36/48)" #1487 daemon prio=5 os_prio=0 tid=0x00007f3c60489800 nid=0xa8c3 in Object.wait() [0x00007f3a9512e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d53aa0df0> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d53af3d70> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5d758> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1486 daemon prio=5 os_prio=0 tid=0x00007f3ba4125000 nid=0xa8c2 in Object.wait() [0x00007f3b09ab5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5dcc8> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (34/48)" #1485 daemon prio=5 os_prio=0 tid=0x00007f3c60488800 nid=0xa8c1 in Object.wait() [0x00007f3a9fc4b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5c642e28> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5c68f788> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5e0e8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1484 daemon prio=5 os_prio=0 tid=0x00007f3bcc11d800 nid=0xa8c0 in Object.wait() [0x00007f3b503c2000]
  java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5e6d8> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (32/48)" #1483 daemon prio=5 os_prio=0 tid=0x00007f3c60488000 nid=0xa8bf in Object.wait() [0x00007f3abfb89000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d53aebef8> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d53b27098> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5eaf8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1482 daemon prio=5 os_prio=0 tid=0x00007f3c7c02a000 nid=0xa8be in Object.wait() [0x00007f3b502c1000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5f068> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (30/48)" #1481 daemon prio=5 os_prio=0 tid=0x00007f3c60adb800 nid=0xa8bd in Object.wait() [0x00007f3a88f14000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d53068458> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5307f990> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5f500> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1480 daemon prio=5 os_prio=0 tid=0x00007f3bc8007800 nid=0xa8bc in Object.wait() [0x00007f3b0c0db000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c5fa70> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (28/48)" #1479 daemon prio=5 os_prio=0 tid=0x00007f3c60adb000 nid=0xa8bb in Object.wait() [0x00007f3b504c3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5c1da5a0> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5c1ef898> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c5fe90> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1478 daemon prio=5 os_prio=0 tid=0x00007f4734025000 nid=0xa8ba in Object.wait() [0x00007f3b506c5000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c60400> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (26/48)" #1477 daemon prio=5 os_prio=0 tid=0x00007f3c60555800 nid=0xa8b9 in Object.wait() [0x00007f3c262ea000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d663c8910> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d663d7c80> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c608a0> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1476 daemon prio=5 os_prio=0 tid=0x00007f3b24004800 nid=0xa8b8 in Object.wait() [0x00007f3a9fd4d000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c60e10> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (24/48)" #1475 daemon prio=5 os_prio=0 tid=0x00007f3c603bc000 nid=0xa8b7 in Object.wait() [0x00007f3a7fde1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5cbd8a58> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5cbe2a98> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c61230> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1474 daemon prio=5 os_prio=0 tid=0x00007f46f812a800 nid=0xa8b6 in Object.wait() [0x00007f3b0c5e0000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c617a0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (22/48)" #1473 daemon prio=5 os_prio=0 tid=0x00007f3c60091000 nid=0xa8b5 in Object.wait() [0x00007f3a9fb4b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5cd8ee68> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5cda77e0> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c61cc8> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1472 daemon prio=5 os_prio=0 tid=0x00007f3bd017a000 nid=0xa8b4 in Object.wait() [0x00007f3b13ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c62238> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (20/48)" #1471 daemon prio=5 os_prio=0 tid=0x00007f3c6029a000 nid=0xa8b3 in Object.wait() [0x00007f3b501c0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d5c03c330> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d5c03ee48> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c62658> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"CloseableReaperThread" #1470 daemon prio=5 os_prio=0 tid=0x00007f3b2815e000 nid=0xa8b2 in Object.wait() [0x00007f3c24ed7000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d94c62d28> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:145)

"phase 1 (18/48)" #1469 daemon prio=5 os_prio=0 tid=0x00007f3c60014000 nid=0xa8b1 in Object.wait() [0x00007f3ab126c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:168)
              - locked <0x00007f3d94ce3290> (a java.util.ArrayDeque)
              at org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:138)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:131)
              - locked <0x00007f3d94d13078> (a org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer)
              at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:88)
              at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:86)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:72)
              at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:39)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:37)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:422)
              at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:407)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:797)
              at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:775)
              at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38)
              at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:185)
              - locked <0x00007f3d94c63148> (a java.lang.Object)
              at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
              at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
              at java.lang.Thread.run(Thread.java:745)

"flink-akka.actor.default-dispatcher-26" #1468 daemon prio=5 os_prio=0 tid=0x00007f3c58875800 nid=0xa872 waiting on condition [0x00007f3a8db28000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-25" #1467 daemon prio=5 os_prio=0 tid=0x00007f3c74a05000 nid=0xa870 waiting on condition [0x00007f3a8ca25000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"nioEventLoopGroup-3-10" #530 prio=10 os_prio=0 tid=0x00007f3bac292800 nid=0x9f3f runnable [0x00007f3a7d6c1000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d4e5f20> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d4e5f08> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d4e5e90> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-11" #531 prio=10 os_prio=0 tid=0x00007f3ba4181000 nid=0x9f3e runnable [0x00007f3a7d7c2000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3dc3e9c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3dc3e9b0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3dc3e938> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-2" #522 prio=10 os_prio=0 tid=0x00007f3bcc115000 nid=0x9f3d runnable [0x00007f3a7d8c3000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d9cbc10> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d9cbbf8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d9cbb80> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-3" #523 prio=10 os_prio=0 tid=0x00007f3b9c298000 nid=0x9f3c runnable [0x00007f3a7d9c4000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d4e62d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d4e62b8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d4e6240> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-4" #524 prio=10 os_prio=0 tid=0x00007f3bc42ac000 nid=0x9f3b runnable [0x00007f3a7dac5000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d4e6680> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d4e6668> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d4e65f0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-5" #525 prio=10 os_prio=0 tid=0x00007f3bbc142000 nid=0x9f3a runnable [0x00007f3a7dbc6000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d9cbe08> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d9cbdf0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d9cbd78> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-6" #526 prio=10 os_prio=0 tid=0x00007f3bb4138800 nid=0x9f39 runnable [0x00007f3a7dcc7000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3dc3ebc0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3dc3eba8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3dc3eb30> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-7" #527 prio=10 os_prio=0 tid=0x00007f3b8c134800 nid=0x9f38 runnable [0x00007f3a7ddc8000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d4e6a30> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d4e6a18> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d4e69a0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-8" #528 prio=10 os_prio=0 tid=0x00007f3b942e5000 nid=0x9f37 runnable [0x00007f3a7dec9000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d9cc000> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d9cbfe8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d9cbf70> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-9" #529 prio=10 os_prio=0 tid=0x00007f3b842a3800 nid=0x9f36 runnable [0x00007f3a7dfca000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3dc3edb8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3dc3eda0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3dc3ed28> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-3-1" #521 prio=10 os_prio=0 tid=0x00007f3b8000c000 nid=0x9f35 runnable [0x00007f3a7e0cb000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d3d9cc1f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d3d9cc1e0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d3d9cc168> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-404-usw1-cl1-ms-mdb11:27017" #574 daemon prio=5 os_prio=0 tid=0x00007f3c7412a000 nid=0x9f34 waiting on condition [0x00007f3a7e1cc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e6ce0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d4e6d40> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-22" #573 daemon prio=5 os_prio=0 tid=0x00007f3c74127800 nid=0x9f33 waiting on condition [0x00007f3a7e2cd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3ef78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-403-usw1-cl1-ms-mdb10:27017" #572 daemon prio=5 os_prio=0 tid=0x00007f3c74125800 nid=0x9f32 waiting on condition [0x00007f3a7e3ce000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cc2f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d9cc358> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-21" #571 daemon prio=5 os_prio=0 tid=0x00007f3c749b5800 nid=0x9f31 waiting on condition [0x00007f3a7e4cf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e7198> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-402-usw1-cl1-ms-mdb9:27017" #570 daemon prio=5 os_prio=0 tid=0x00007f3c749b3800 nid=0x9f30 waiting on condition [0x00007f3a7e5d0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cc540> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d9cc5a0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-20" #569 daemon prio=5 os_prio=0 tid=0x00007f3c749b1800 nid=0x9f2f waiting on condition [0x00007f3a7e6d1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3f0e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-401-usw1-cl1-ms-mdb8:27017" #568 daemon prio=5 os_prio=0 tid=0x00007f3c74090800 nid=0x9f2e waiting on condition [0x00007f3a7e7d2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e73f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d4e7458> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-19" #567 daemon prio=5 os_prio=0 tid=0x00007f3c7408e800 nid=0x9f2d waiting on condition [0x00007f3a7e8d3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cc848> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-400-usw1-cl1-ms-mdb7:27017" #566 daemon prio=5 os_prio=0 tid=0x00007f3c7432a000 nid=0x9f2c waiting on condition [0x00007f3a7e9d4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3f198> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3f1f8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-18" #565 daemon prio=5 os_prio=0 tid=0x00007f3c74328000 nid=0x9f2b waiting on condition [0x00007f3a7ead5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e78b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-289-usw1-cl1-ms-mdb2:27017" #564 daemon prio=5 os_prio=0 tid=0x00007f3c7495c800 nid=0x9f2a waiting on condition [0x00007f3a7ebd6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cc8f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d9cc958> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-17" #563 daemon prio=5 os_prio=0 tid=0x00007f3c74302000 nid=0x9f29 waiting on condition [0x00007f3a7ecd7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e7bd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-22-usw1-cl1-ms-mdb3:27017" #562 daemon prio=5 os_prio=0 tid=0x00007f3c74300800 nid=0x9f28 waiting on condition [0x00007f3a7edd8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3f3e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3f440> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-16" #561 daemon prio=5 os_prio=0 tid=0x00007f3c74061000 nid=0x9f27 waiting on condition [0x00007f3a7eed9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9ccc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-20-usw1-cl1-ms-mdb1:27017" #560 daemon prio=5 os_prio=0 tid=0x00007f3c7405f800 nid=0x9f26 waiting on condition [0x00007f3a7efda000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3f628> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3f688> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-15" #559 daemon prio=5 os_prio=0 tid=0x00007f3c74142000 nid=0x9f25 waiting on condition [0x00007f3a7f0db000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e7ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-185-usw1-cl1-ms-mdb6:27017" #558 daemon prio=5 os_prio=0 tid=0x00007f3c74140800 nid=0x9f24 waiting on condition [0x00007f3a7f1dc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3f870> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3f8d0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-14" #557 daemon prio=5 os_prio=0 tid=0x00007f3c74a17800 nid=0x9f23 waiting on condition [0x00007f3a7f2dd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9ccd70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-184-usw1-cl1-ms-mdb5:27017" #556 daemon prio=5 os_prio=0 tid=0x00007f3c74a16000 nid=0x9f22 waiting on condition [0x00007f3a7f3de000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e8150> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d4e81b0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-13" #555 daemon prio=5 os_prio=0 tid=0x00007f3c74424000 nid=0x9f21 waiting on condition [0x00007f3a7f4df000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d273508> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e01', description='null'}-prod-183-usw1-cl1-ms-mdb4:27017" #554 daemon prio=5 os_prio=0 tid=0x00007f3c7467d800 nid=0x9f20 waiting on condition [0x00007f3a7f5e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3fab8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3fb18> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-12" #553 daemon prio=5 os_prio=0 tid=0x00007f3c74368000 nid=0x9f1f waiting on condition [0x00007f3a7f6e1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e8608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"pool-3-thread-1" #520 daemon prio=5 os_prio=0 tid=0x00007f3c74420800 nid=0x9f1e waiting on condition [0x00007f3a7ffe3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cce20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-404-usw1-cl1-ms-mdb11:27017" #519 daemon prio=5 os_prio=0 tid=0x00007f3c747a7000 nid=0x9f1d waiting on condition [0x00007f3a80505000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc3fd00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc3fd60> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-11" #518 daemon prio=5 os_prio=0 tid=0x00007f3c747a5000 nid=0x9f1c waiting on condition [0x00007f3a80606000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc40008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-403-usw1-cl1-ms-mdb10:27017" #517 daemon prio=5 os_prio=0 tid=0x00007f3c747a3800 nid=0x9f1b waiting on condition [0x00007f3a80707000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e8868> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d4e88c8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-10" #516 daemon prio=5 os_prio=0 tid=0x00007f3c747dc800 nid=0x9f19 waiting on condition [0x00007f3a80808000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cd018> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-402-usw1-cl1-ms-mdb9:27017" #515 daemon prio=5 os_prio=0 tid=0x00007f3c747da800 nid=0x9f15 waiting on condition [0x00007f3a80909000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc400b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc40118> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-9" #514 daemon prio=5 os_prio=0 tid=0x00007f3c74094800 nid=0x9f14 waiting on condition [0x00007f3a80a0a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d273828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-401-usw1-cl1-ms-mdb8:27017" #513 daemon prio=5 os_prio=0 tid=0x00007f3c74092800 nid=0x9f13 waiting on condition [0x00007f3a80b0b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e8c60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d4e8cc0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-8" #512 daemon prio=5 os_prio=0 tid=0x00007f3c74a1c000 nid=0x9f12 waiting on condition [0x00007f3a80c0c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cd188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-400-usw1-cl1-ms-mdb7:27017" #511 daemon prio=5 os_prio=0 tid=0x00007f3c74a1a800 nid=0x9f11 waiting on condition [0x00007f3a80d0d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc40300> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc40360> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-7" #510 daemon prio=5 os_prio=0 tid=0x00007f3c74a18800 nid=0x9f10 waiting on condition [0x00007f3a80e0e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e9118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-289-usw1-cl1-ms-mdb2:27017" #509 daemon prio=5 os_prio=0 tid=0x00007f3c7477b800 nid=0x9f0f waiting on condition [0x00007f3a80f0f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d273a88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d273ae8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-6" #508 daemon prio=5 os_prio=0 tid=0x00007f3c7477a000 nid=0x9f0e waiting on condition [0x00007f3a81010000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cd2f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-22-usw1-cl1-ms-mdb3:27017" #507 daemon prio=5 os_prio=0 tid=0x00007f3c74948800 nid=0x9f0d waiting on condition [0x00007f3a81111000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc40548> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3dc405a8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-5" #506 daemon prio=5 os_prio=0 tid=0x00007f3c74947000 nid=0x9f0c waiting on condition [0x00007f3af3368000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e9438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-20-usw1-cl1-ms-mdb1:27017" #505 daemon prio=5 os_prio=0 tid=0x00007f3c74946000 nid=0x9f07 waiting on condition [0x00007f3afc58b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d273e80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d273ee0> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-4" #504 daemon prio=5 os_prio=0 tid=0x00007f3c743bd000 nid=0x9f06 waiting on condition [0x00007f3afc68c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc40850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-185-usw1-cl1-ms-mdb6:27017" #503 daemon prio=5 os_prio=0 tid=0x00007f3c743bc000 nid=0x9f05 waiting on condition [0x00007f3afc78d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cd3a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d9cd408> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-3" #502 daemon prio=5 os_prio=0 tid=0x00007f3c743bb000 nid=0x9f04 waiting on condition [0x00007f3afc48a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d4e9758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-184-usw1-cl1-ms-mdb5:27017" #501 daemon prio=5 os_prio=0 tid=0x00007f3c746e1000 nid=0x9f03 waiting on condition [0x00007f3afbf85000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d274278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d2742d8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-2" #500 daemon prio=5 os_prio=0 tid=0x00007f3c746e0000 nid=0x9f02 waiting on condition [0x00007f3afc288000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3dc409c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29122d65b0b983b549e00', description='null'}-prod-183-usw1-cl1-ms-mdb4:27017" #499 daemon prio=5 os_prio=0 tid=0x00007f3c7404f000 nid=0x9f01 waiting on condition [0x00007f3afc389000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d9cd5f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d3d9cd650> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" #498 daemon prio=5 os_prio=0 tid=0x00007f3c7436a000 nid=0x9f00 waiting on condition [0x00007f3afb57b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d3d274580> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 10" #82 daemon prio=5 os_prio=0 tid=0x00007f3c08191000 nid=0x9ef0 runnable [0x00007f3afbe84000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d183414f0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18340280> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18340140> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 7" #63 daemon prio=5 os_prio=0 tid=0x00007f3b9428c800 nid=0x9ee0 runnable [0x00007f3afb278000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182cf190> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182cdf20> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182cdde0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 6" #62 daemon prio=5 os_prio=0 tid=0x00007f3b9428b800 nid=0x9edf runnable [0x00007f3afb177000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182c9660> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182c83f0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182c82b0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 9" #81 daemon prio=5 os_prio=0 tid=0x00007f3c08171000 nid=0x9ecf runnable [0x00007f3afb076000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1833b9c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1833a750> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1833a610> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 8" #80 daemon prio=5 os_prio=0 tid=0x00007f3c08170000 nid=0x9ec7 runnable [0x00007f3afc086000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18335e90> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18334c20> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18334ae0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 5" #61 daemon prio=5 os_prio=0 tid=0x00007f3c00162000 nid=0x9ec4 runnable [0x00007f3afc187000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182c3b30> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182c28c0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182c2780> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 4" #60 daemon prio=5 os_prio=0 tid=0x00007f3c00158800 nid=0x9ec3 runnable [0x00007f3afba80000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182be000> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182bcd90> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182bcc50> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 3" #59 daemon prio=5 os_prio=0 tid=0x00007f3c9023a000 nid=0x9ec1 runnable [0x00007f3afb97f000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182b84d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182b7260> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182b7120> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 2" #58 daemon prio=5 os_prio=0 tid=0x00007f3c90262000 nid=0x9ec0 runnable [0x00007f3afb47a000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182b29a0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182b1730> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182b15f0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 1" #57 daemon prio=5 os_prio=0 tid=0x00007f3b482f2800 nid=0x9eb1 runnable [0x00007f3afb77d000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182ace70> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182abc00> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182abac0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Client (6141) Thread 0" #56 daemon prio=5 os_prio=0 tid=0x00007f3b482eb800 nid=0x9eb0 runnable [0x00007f3afbb81000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d182a7340> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182a60d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d182a5f90> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 7" #79 daemon prio=5 os_prio=0 tid=0x00007f3c0816e800 nid=0x9e49 runnable [0x00007f3afd990000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18330360> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1832f0f0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1832efb0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 6" #78 daemon prio=5 os_prio=0 tid=0x00007f3c0816c800 nid=0x9e48 runnable [0x00007f3afea92000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1832a830> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d183295c0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18329480> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 5" #77 daemon prio=5 os_prio=0 tid=0x00007f3c0816b000 nid=0x9e40 runnable [0x00007f3affe95000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18324d00> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18323a90> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18323950> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-1" #342 daemon prio=5 os_prio=0 tid=0x00007f3bd4211000 nid=0x9e15 waiting on condition [0x00007f3b02ebb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d35ab1848> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29104d65b0b983b549dd4', description='null'}-prod-362-usw1-cl1-mgmt-mdb5:27017" #341 daemon prio=5 os_prio=0 tid=0x00007f3bd420e800 nid=0x9e14 waiting on condition [0x00007f3b02fbc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d35ab1a70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d35ab1a98> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29104d65b0b983b549dd4', description='null'}-prod-27-usw1-cl1-mgmt-mdb2:27017" #340 daemon prio=5 os_prio=0 tid=0x00007f3bd420d000 nid=0x9e13 waiting on condition [0x00007f3b030bd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d35ab1d00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d35ab1d28> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"cluster-ClusterId{value='58b29104d65b0b983b549dd4', description='null'}-prod-26-usw1-cl1-mgmt-mdb1:27017" #339 daemon prio=5 os_prio=0 tid=0x00007f3bd420b000 nid=0x9e12 waiting on condition [0x00007f3b031be000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d35ab1f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x00007f3d35ab1fb8> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)

"flink-akka.actor.default-dispatcher-24" #338 daemon prio=5 os_prio=0 tid=0x00007f3b74183000 nid=0x9e10 waiting on condition [0x00007f3b03706000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-23" #337 daemon prio=5 os_prio=0 tid=0x00007f3bcc10b000 nid=0x9e0f waiting on condition [0x00007f3b03807000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-22" #336 daemon prio=5 os_prio=0 tid=0x00007f3bb0004800 nid=0x9e0e waiting on condition [0x00007f3b03908000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-21" #335 daemon prio=5 os_prio=0 tid=0x00007f3bc4248000 nid=0x9e0d waiting on condition [0x00007f3b03a09000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-20" #334 daemon prio=5 os_prio=0 tid=0x00007f3b74182800 nid=0x9e0c waiting on condition [0x00007f3b03b0a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-19" #333 daemon prio=5 os_prio=0 tid=0x00007f3bbc112800 nid=0x9e0b waiting on condition [0x00007f3b03c0b000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-18" #332 daemon prio=5 os_prio=0 tid=0x00007f3bbc10f000 nid=0x9e0a waiting on condition [0x00007f3b03d0c000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-17" #331 daemon prio=5 os_prio=0 tid=0x00007f3b84231000 nid=0x9e09 waiting on condition [0x00007f3b03e0d000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-16" #330 daemon prio=5 os_prio=0 tid=0x00007f3b9c229000 nid=0x9e08 waiting on condition [0x00007f3b03f0e000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"BackpressureMonitor" #329 daemon prio=5 os_prio=0 tid=0x00007f3b8c115800 nid=0x9e07 waiting on condition [0x00007f3b0400f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at com.utils.BackpressureMonitor.run(BackpressureMonitor.java:78)

"Flink Netty Server (6141) Thread 4" #76 daemon prio=5 os_prio=0 tid=0x00007f3c08030000 nid=0x99f2 runnable [0x00007f3b098b3000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1831f1d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1831df60> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1831de20> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 3" #75 daemon prio=5 os_prio=0 tid=0x00007f3c0802f000 nid=0x99ef runnable [0x00007f3b09bb6000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d183196a0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18318430> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d183182f0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #263 daemon prio=1 os_prio=0 tid=0x00007f4718004000 nid=0x99b6 waiting on condition [0x00007f3b0f1ee000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
              at java.lang.Thread.run(Thread.java:745)

"flink-akka.actor.default-dispatcher-15" #262 daemon prio=5 os_prio=0 tid=0x00007f3c3c021000 nid=0x99b5 waiting on condition [0x00007f3b0f2ef000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"Flink Netty Server (6141) Thread 2" #74 daemon prio=5 os_prio=0 tid=0x00007f3c0800e800 nid=0x99b4 runnable [0x00007f3b0f3f0000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18313b70> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d18312900> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d183127c0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 1" #73 daemon prio=5 os_prio=0 tid=0x00007f3c0800d800 nid=0x99b3 runnable [0x00007f3b0f4f1000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1830e040> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d1830cdd0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d1830cc90> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Timer-0" #99 daemon prio=5 os_prio=0 tid=0x00007f3c3401d000 nid=0x9897 in Object.wait() [0x00007f3c265ee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.util.TimerThread.mainLoop(Timer.java:552)
              - locked <0x00007f3d36b92f50> (a java.util.TaskQueue)
              at java.util.TimerThread.run(Timer.java:505)

"Hashed wheel timer #1" #35 daemon prio=5 os_prio=0 tid=0x00007f3c18001000 nid=0x9896 waiting on condition [0x00007f3c268ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
              at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at java.lang.Thread.run(Thread.java:745)

"Flink-MetricRegistry-1" #96 prio=5 os_prio=0 tid=0x00007f475efd2800 nid=0x9895 waiting on condition [0x00007f3c26bf0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d36b92698> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"Curator-Framework-0" #95 daemon prio=5 os_prio=0 tid=0x00007f475ef7c000 nid=0x9894 waiting on condition [0x00007f3c26cf1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d366efa20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
              at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
              at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
              at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"main-EventThread" #94 daemon prio=5 os_prio=0 tid=0x00007f475ef5c800 nid=0x9893 waiting on condition [0x00007f3c26df2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d36da8048> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

"main-SendThread(10.0.5.28:2181)" #93 daemon prio=5 os_prio=0 tid=0x00007f475ef86000 nid=0x9892 runnable [0x00007f3c26ef3000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d36b929e8> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d36b929d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d36b92958> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
              at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

"Curator-ConnectionStateManager-0" #92 daemon prio=5 os_prio=0 tid=0x00007f475ef56000 nid=0x9890 waiting on condition [0x00007f3c271f4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d364af0c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
              at org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
              at org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
              at org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"IOManager reader thread #1" #90 daemon prio=5 os_prio=0 tid=0x00007f475e18f800 nid=0x988f waiting on condition [0x00007f3c272f5000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d36926fa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$ReaderThread.run(IOManagerAsync.java:380)

"IOManager writer thread #1" #89 daemon prio=5 os_prio=0 tid=0x00007f475ed4d800 nid=0x988e waiting on condition [0x00007f3c273f6000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d36da8128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:486)

"Flink KvStateServer EventLoop Thread 0" #40 daemon prio=5 os_prio=0 tid=0x00007f475ed4b800 nid=0x988b runnable [0x00007f3c274f7000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d1821a5f0> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d182185b0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18218470> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"Flink Netty Server (6141) Thread 0" #72 daemon prio=5 os_prio=0 tid=0x00007f475ed48000 nid=0x988a runnable [0x00007f3c275f8000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d18308510> (a io.netty.channel.nio.SelectedSelectionKeySet)
              - locked <0x00007f3d183072c0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d18307180> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)

"New I/O server boss #6" #39 daemon prio=5 os_prio=0 tid=0x00007f3c200c0000 nid=0x9888 runnable [0x00007f3c27af9000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aaff9e8> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aaff9d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aaff838> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"New I/O worker #5" #38 daemon prio=5 os_prio=0 tid=0x00007f3c2006c800 nid=0x9887 runnable [0x00007f3c27bfa000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aafa840> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aafa828> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aafa690> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"New I/O worker #4" #37 daemon prio=5 os_prio=0 tid=0x00007f3c2006c000 nid=0x9886 runnable [0x00007f3c27cfb000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aaf8c80> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aaf8c68> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aaf8ad0> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"New I/O boss #3" #36 daemon prio=5 os_prio=0 tid=0x00007f3c2006e800 nid=0x9885 runnable [0x00007f3c27dfc000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aaee1c0> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aaee1a8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aaee010> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"New I/O worker #2" #34 daemon prio=5 os_prio=0 tid=0x00007f3c2002b800 nid=0x9884 runnable [0x00007f3c27efd000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aa68688> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aa68670> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aa684d8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"New I/O worker #1" #33 daemon prio=5 os_prio=0 tid=0x00007f3c2002e800 nid=0x9883 runnable [0x00007f3c27ffe000]
   java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00007f3d2aa5e580> (a sun.nio.ch.Util$3)
              - locked <0x00007f3d2aa5e4c8> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00007f3d2aa5e330> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

"flink-akka.remote.default-remote-dispatcher-7" #32 daemon prio=5 os_prio=0 tid=0x00007f3c3c01c000 nid=0x9882 waiting on condition [0x00007f3c841df000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2ac07608> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.remote.default-remote-dispatcher-6" #31 daemon prio=5 os_prio=0 tid=0x00007f3c30009800 nid=0x9881 waiting on condition [0x00007f3c844e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2ac07608> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-5" #30 daemon prio=5 os_prio=0 tid=0x00007f3c3c009800 nid=0x9880 waiting on condition [0x00007f3c845e1000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-4" #29 daemon prio=5 os_prio=0 tid=0x00007f475e6a2000 nid=0x987f waiting on condition [0x00007f3c846e2000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-3" #28 daemon prio=5 os_prio=0 tid=0x00007f475e69f800 nid=0x987e waiting on condition [0x00007f3c847e3000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-2" #27 daemon prio=5 os_prio=0 tid=0x00007f475e68e000 nid=0x987d waiting on condition [0x00007f3c848e4000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2aca8da8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
              at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-scheduler-1" #26 daemon prio=5 os_prio=0 tid=0x00007f475e5eb000 nid=0x987b sleeping[0x00007f3c851ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:231)
              at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:411)
              at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
              at java.lang.Thread.run(Thread.java:745)

"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f475db9d000 nid=0x9874 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #19 daemon prio=9 os_prio=0 tid=0x00007f475db99800 nid=0x9873 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #18 daemon prio=9 os_prio=0 tid=0x00007f475db98000 nid=0x9872 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #17 daemon prio=9 os_prio=0 tid=0x00007f475db95800 nid=0x9871 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #16 daemon prio=9 os_prio=0 tid=0x00007f475db93000 nid=0x9870 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #15 daemon prio=9 os_prio=0 tid=0x00007f475db91000 nid=0x986f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #14 daemon prio=9 os_prio=0 tid=0x00007f475db8f000 nid=0x986e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #13 daemon prio=9 os_prio=0 tid=0x00007f475db8d000 nid=0x986d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #12 daemon prio=9 os_prio=0 tid=0x00007f475db8a800 nid=0x986c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #11 daemon prio=9 os_prio=0 tid=0x00007f475db80800 nid=0x986b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #10 daemon prio=9 os_prio=0 tid=0x00007f475db7e800 nid=0x986a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #9 daemon prio=9 os_prio=0 tid=0x00007f475db7c000 nid=0x9869 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #8 daemon prio=9 os_prio=0 tid=0x00007f475db7a000 nid=0x9868 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x00007f475db78000 nid=0x9867 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x00007f475db74000 nid=0x9866 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f475db66800 nid=0x9865 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f475db65000 nid=0x9864 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f475db33800 nid=0x9863 in Object.wait() [0x00007f3c9c34b000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00007f3d2487c2e0> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f475db2f000 nid=0x9862 in Object.wait() [0x00007f3c9c44c000]
   java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
              - locked <0x00007f3d2483cb88> (a java.lang.ref.Reference$Lock)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f475c014000 nid=0x9841 waiting on condition [0x00007f4762e5e000]
   java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f3d2b103cd0> (a java.util.concurrent.CountDownLatch$Sync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
              at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:819)
              at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:788)
              at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
              at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
              at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
              at scala.concurrent.Await$.ready(package.scala:86)
              at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:644)
              at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:645)
              at org.apache.flink.runtime.taskmanager.TaskManager$.runTaskManager(TaskManager.scala:1805)
              at org.apache.flink.runtime.taskmanager.TaskManager$.selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1639)
              at org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1548)
              at org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1546)
              at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
              at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
              at org.apache.flink.runtime.taskmanager.TaskManager$.main(TaskManager.scala:1546)
              at org.apache.flink.runtime.taskmanager.TaskManager.main(TaskManager.scala)

"VM Thread" os_prio=0 tid=0x00007f475db27000 nid=0x9861 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c02c000 nid=0x9842 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c02e000 nid=0x9843 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c02f800 nid=0x9844 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c031000 nid=0x9845 runnable

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c033000 nid=0x9846 runnable

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c034800 nid=0x9847 runnable

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c036800 nid=0x9848 runnable

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c038000 nid=0x9849 runnable

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c03a000 nid=0x984a runnable

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c03b800 nid=0x984b runnable

"Gang worker#10 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c03d800 nid=0x984c runnable

"Gang worker#11 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c03f000 nid=0x984d runnable

"Gang worker#12 (Parallel GC Threads)" os_prio=0 tid=0x00007f475c041000 nid=0x984e runnable

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f475c072000 nid=0x985d runnable

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f475c074000 nid=0x985e runnable

"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f475c075800 nid=0x985f runnable

"Gang worker#2 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f475c077800 nid=0x9860 runnable

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f475c05b000 nid=0x985c runnable

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f475c059800 nid=0x985b runnable

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f475c057800 nid=0x985a runnable

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f475c055800 nid=0x9859 runnable

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f475c054000 nid=0x9858 runnable

"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f475c052000 nid=0x9857 runnable

"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f475c050800 nid=0x9856 runnable

"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f475c04e800 nid=0x9855 runnable

"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f475c04d000 nid=0x9854 runnable

"G1 Concurrent Refinement Thread#9" os_prio=0 tid=0x00007f475c04b000 nid=0x9853 runnable

"G1 Concurrent Refinement Thread#10" os_prio=0 tid=0x00007f475c049800 nid=0x9852 runnable

"G1 Concurrent Refinement Thread#11" os_prio=0 tid=0x00007f475c047800 nid=0x9851 runnable

"G1 Concurrent Refinement Thread#12" os_prio=0 tid=0x00007f475c046000 nid=0x9850 runnable

"G1 Concurrent Refinement Thread#13" os_prio=0 tid=0x00007f475c044000 nid=0x984f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f475dba0000 nid=0x9875 waiting on condition

JNI global references: 12990


From: Stephan Ewen [mailto:sewen@apache.org]
Sent: Friday, February 24, 2017 2:36 PM
To: user@flink.apache.org
Subject: Re: Flink checkpointing gets stuck

Hi Shai!

It looks to me like failures are caused by the fact that the Azure File System behaves semantically a slight bit different than what Flink expects and that may throw it off-guard.
Here are some thoughts:

  - Flink 1.1.x behaves differently than 1.2 when it comes to expectations filesystem metadata consistency. 1.2 expects a little less, as a result of some patchers to better support S3.

  - I tried to write up what expectations we have for filesystems. You can check if you see and points where the Azure Block Store diverges in its semantics.
    The expectations are in the latest master in the docs of the "FileSystem" class: https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink%2Fblob%2Fmaster%2Fflink-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcore%2Ffs%2FFileSystem.java&data=02%7C01%7CShai.Kaplan%40microsoft.com%7Cfb05f5e7182c43ae68d008d45cb1abd7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636235365543234360&sdata=vfwEj47lhteTzHYLE6JzDwMjLF5uzp6Wqyfp7Qekc%2Bo%3D&reserved=0>
    Below is a quite with relevant parts highlighted.


Completely aside from that, these failures should not make the checkpoint get stuck. If you can attach a tread dump, we can look into what it is not at least "failing well".

Greetings,
Stephan

=============


 *

 * <h2>Data Persistence Contract</h2>

 *

 * The FileSystem's {@link FSDataOutputStream output streams} are used to persistently store data,

 * both for results of streaming applications and for fault tolerance and recovery. It is therefore

 * crucial that the persistence semantics of these streams are well defined.

 *

 * <h3>Definition of Persistence Guarantees</h3>

 *

 * Data written to an output stream is considered persistent, if two requirements are met:

 *

 * <ol>

 *     <li><b>Visibility Requirement:</b> It must be guaranteed that all other processes, machines,

 *     virtual machines, containers, etc. that are able to access the file see the data consistently

 *     when given the absolute file path. This requirement is similar to the <i>close-to-open</i>

 *     semantics defined by POSIX, but restricted to the file itself (by its absolute path).</li>

 *

 *     <li><b>Durability Requirement:</b> The file system's specific durability/persistence requirements

 *     must be met. These are specific to the particular file system. For example the

 *     {@link LocalFileSystem} does not provide any durability guarantees for crashes of both

 *     hardware and operating system, while replicated distributed file systems (like HDFS)

 *     typically guarantee durability in the presence of at most <i>n</i> concurrent node failures,

 *     where <i>n</i> is the replication factor.</li>

 * </ol>

 *

 * <p>Updates to the file's parent directory (such that the file shows up when

 * listing the directory contents) are __not__ required to be complete for the data in the file stream

 * to be considered persistent. This relaxation is important for file systems where updates to

 * directory contents are only eventually consistent.

 *

 * <p>The {@link FSDataOutputStream} has to guarantee data persistence for the written bytes

 * once the call to {@link FSDataOutputStream#close()} returns.

 *

 * <h3>Examples</h3>

 *

 * <ul>

 *     <li>For <b>fault-tolerant distributed file systems</b>, data is considered persistent once

 *     it has been received and acknowledged by the file system, typically by having been replicated

 *     to a quorum of machines (<i>durability requirement</i>). In addition the absolute file path

 *     must be visible to all other machines that will potentially access the file (<i>visibility

 *     requirement</i>).

 *

 *     <p>Whether data has hit non-volatile storage on the storage nodes depends on the specific

 *     guarantees of the particular file system.

 *

 *     <p>The metadata updates to the file's parent directory are not required to have reached

 *     a consistent state. It is permissible that some machines see the file when listing the parent

 *     directory's contents while others do not, as long as access to the file by its absolute path

 *     is possible on all nodes.</li>

 *

 *     <li>A <b>local file system</b> must support the POSIX <i>close-to-open</i> semantics.

 *     Because the local file system does not have any fault tolerance guarantees, no further

 *     requirements exist.

 *

 *     <p>The above implies specifically that data may still be in the OS cache when considered

 *     persistent from the local file system's perspective. Crashes that cause the OS cache to loose

 *     data are considered fatal to the local machine and are not covered by the local file system's

 *     guarantees as defined by Flink.

 *

 *     <p>That means that computed results, checkpoints, and savepoints that are written only to

 *     the local filesystem are not guaranteed to be recoverable from the local machine's failure,

 *     making local file systems unsuitable for production setups.</li>

 * </ul>

 *

 * <h2>Updating File Contents</h2>

 *

 * Many file systems either do not support overwriting contents of existing files at all, or do

 * not support consistent visibility of the updated contents in that case. For that reason,

 * Flink's FileSystem does not support appending to existing files, or seeking within output streams

 * so that previously written data could be overwritten.

 *

 * <h2>Overwriting Files</h2>

 *

 * Overwriting files is in general possible. A file is overwritten by deleting it and creating

 * a new file. However, certain filesystems cannot make that change synchronously visible

 * to all parties that have access to the file.

 * For example <a href="https://aws.amazon.com/documentation/s3/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fdocumentation%2Fs3%2F&data=02%7C01%7CShai.Kaplan%40microsoft.com%7Cfb05f5e7182c43ae68d008d45cb1abd7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636235365543244368&sdata=EEmcQp3Ynn0ubNfmoz%2BdhEwkqLmuY0xb7TFnSXgf0gE%3D&reserved=0>">Amazon S3</a> guarantees only

 * <i>eventual consistency</i> in the visibility of the file replacement: Some machines may see

 * the old file, some machines may see the new file.

 *

 * <p>To avoid these consistency issues, the implementations of failure/recovery mechanisms in

 * Flink strictly avoid writing to the same file path more than once.

 *

 * <h2>Thread Safety</h2>

 *

 * Implementations of {@code FileSystem} must be thread-safe: The same instance of FileSystem

 * is frequently shared across multiple threads in Flink and must be able to concurrently

 * create input/output streams and list file metadata.

 *

 * <p>The {@link FSDataOutputStream} and {@link FSDataOutputStream} implementations are strictly

 * <b>not thread-safe</b>. Instances of the streams should also not be passed between threads

 * in between read or write operations, because there are no guarantees about the visibility of

 * operations across threads (many operations do not create memory fences).



On Thu, Feb 23, 2017 at 9:31 PM, Robert Metzger <rm...@apache.org>> wrote:
Hi Shai,

I think we don't have so many users running Flink on Azure. Maybe you are the first to put some heavy load onto that infrastructure using Flink.
I would guess that your problems are caused by the same root cause, just the way the job is being cancelled is a bit different based on what is happening.
For the "The specified blob does not exist". Are there config options for the "NativeAzureFileSystem" that allow you to enable retries?
Maybe its an "eventual consistency" issue of the underlying file store. (Flink is creating a directory against endpointA, and endpointB has not synced yet?).

Can you check for this checkpoint the details page:
763             241/241         11:12:50        11:28:38                        15m 48s

It contains a breakdown what took so long.
Was it the alignment time? the async time for the state to be snapshotted?

If you want, you can also contact me privately and we do a Hangout session to look at the Flink UI together.

Regards,
Robert





On Thu, Feb 23, 2017 at 8:59 AM, Shai Kaplan <Sh...@microsoft.com>> wrote:
And now it's happening again

-----Original Message-----
From: Shai Kaplan [mailto:Shai.Kaplan@microsoft.com<ma...@microsoft.com>]
Sent: Wednesday, February 22, 2017 12:02 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink checkpointing gets stuck

I changed the checkpoint interval to 30 minutes, and also switched RocksDB predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay. The problem hasn't exactly occurred since yesterday, but perhaps it just takes it longer to happen again because the checkpoints are much less frequent now.
I'm now pretty sure that the " Checkpoint Coordinator is suspending" is not related to the other issue, because now I had a checkpoint that failed because of that but then the next ones succeeded. I apologize for diverging from the topic, but I think it's worth mentioning too, the exception was "Could not materialize checkpoint" caused by "Could not flush and close the file system output stream" caused by "com.microsoft.azure.storage.StorageException: The specified blob does not exist". What could cause that? A temporary failure in I/O with Azure blobs?

Anyway, back to the topic, I didn't have checkpoints timing out, but I did have one checkpoint that took significantly longer. This is what the checkpoints history look like right now:

ID      Status  Acknowledged    Trigger Time    Latest Acknowledgement  End to End Duration     State Size      Buffered During Alignment
764             241/241         11:42:50        11:43:56                        1m 6s                   3.15 GB         30.4 MB
763             241/241         11:12:50        11:28:38                        15m 48s                 3.11 GB         13.4 MB
762             241/241         10:42:50        10:43:57                        1m 7s                   3.08 GB         12.8 MB
761             241/241         10:12:50        10:13:58                        1m 8s                   3.03 GB         5.52 MB
760             241/241         9:42:50         9:43:55                         1m 5s                   2.99 GB         607 KB
759             241/241         9:12:50         9:13:52                         1m 1s                   2.94 GB         0 B
758             241/241         8:42:50         8:43:46                         56s                     2.90 GB         5.55 MB
757             241/241         8:12:50         8:14:20                         1m 30s                  2.85 GB         0 B
756             121/241         7:41:49         7:42:09                         22s                     280 MB  0 B
755             241/241         7:11:49         7:12:44                         55s                     2.81 GB         30 B

The status wasn't copied well, but they all succeeded except for #756, which failed for the reason I mentioned above.
As you can see, checkpoint #763 took a lot longer for no apparent reason, so I'm guessing it's the same thing that caused the checkpoints to time out at 30 minutes when they were saved every 10 seconds, only now it's less severe because the load is much lower.
Any thoughts on what could cause that?

-----Original Message-----
From: Ufuk Celebi [mailto:uce@apache.org<ma...@apache.org>]
Sent: Tuesday, February 21, 2017 4:54 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink checkpointing gets stuck

Hey Shai!

Thanks for reporting this.

It's hard to tell what causes this from your email, but could you check the checkpoint interface
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fmonitoring%2Fcheckpoint_monitoring.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C1cdb8bde8ee843676a0008d45a6984b7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232856631405599&sdata=Ukf1%2Bahi00NVb3QVrTqUggYOSldheqvgMkJLsCzUWVc%3D&reserved=0)
and report how much progress the checkpoints make before timing out?

The "Checkpoint Coordinator is suspending" message indicates that the job failed and the checkpoint coordinator is shut down because of that. Can you check the TaskManager and JobManager logs if other errors are reported? Feel free to share them. Then I could help with going over them.

– Ufuk


On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com>> wrote:
> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval.
> After some running time (minutes-hours) Flink fails to save
> checkpoints, and stops processing records (I'm not sure if the
> checkpointing failure is the cause of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start
> failing due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job
> restarted), the job is recovered from the last successful checkpoint
> (the state size continues to grow, so it's probably not the reason for
> the failure), advances somewhat, saves some more checkpoints, and then
> enters the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due
> to "Checkpoint Coordinator is suspending.", so it might be an
> indicator for the cause of the problem, but looking into Flink's code
> I can't see how a running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob
> Store, using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?

Re: Flink checkpointing gets stuck

Posted by Stephan Ewen <se...@apache.org>.

Hi Shai!

It looks to me like failures are caused by the fact that the Azure File
System behaves semantically a slight bit different than what Flink expects
and that may throw it off-guard.
Here are some thoughts:

  - Flink 1.1.x behaves differently than 1.2 when it comes to expectations
filesystem metadata consistency. 1.2 expects a little less, as a result of
some patchers to better support S3.

  - I tried to write up what expectations we have for filesystems. You can
check if you see and points where the Azure Block Store diverges in its
semantics.
    The expectations are in the latest master in the docs of the
"FileSystem" class: https://github.com/apache/flink/blob/master/flink-core/
src/main/java/org/apache/flink/core/fs/FileSystem.java
    Below is a quite with relevant parts highlighted.

Completely aside from that, these failures should not make the checkpoint
get stuck. If you can attach a tread dump, we can look into what it is not
at least "failing well".

Greetings,
Stephan

=============

 *
 * <h2>Data Persistence Contract</h2>
 *
 * The FileSystem's {@link FSDataOutputStream output streams} are used
to persistently store data,
 * both for results of streaming applications and for fault tolerance
and recovery. It is therefore
 * crucial that the persistence semantics of these streams are well defined.
 *
 * <h3>Definition of Persistence Guarantees</h3>
 *
 * Data written to an output stream is considered persistent, if two
requirements are met:
 *
 * <ol>
 *     <li><b>Visibility Requirement:</b> It must be guaranteed that
all other processes, machines,
 *     virtual machines, containers, etc. that are able to access the
file see the data consistently
 *     when given the absolute file path. *This requirement is similar
to the <i>close-to-open</i>
 *     semantics defined by POSIX, but restricted to the file itself
(by its absolute path)*.</li>
 *
 *     <li><b>Durability Requirement:</b> The file system's specific
durability/persistence requirements
 *     must be met. These are specific to the particular file system.
For example the
 *     {@link LocalFileSystem} does not provide any durability
guarantees for crashes of both
 *     hardware and operating system, while replicated distributed
file systems (like HDFS)
 *     typically guarantee durability in the presence of at most
<i>n</i> concurrent node failures,
 *     where <i>n</i> is the replication factor.</li>
 * </ol>
 *
 * <p>*Updates to the file's parent directory (such that the file shows up when
 * listing the directory contents) are __not__ required to be
complete* for the data in the file stream
 * to be considered persistent. This relaxation is important for file
systems where updates to
 * directory contents are only eventually consistent.
 *
 * <p>*The {@link FSDataOutputStream} has to guarantee data
persistence for the written bytes
 * once the call to {@link FSDataOutputStream#close()} returns.*
 *
 * <h3>Examples</h3>
 *
 * <ul>
 *     <li>For <b>fault-tolerant distributed file systems</b>, data is
considered persistent once
 *     it has been received and acknowledged by the file system,
typically by having been replicated
 *     to a quorum of machines (<i>durability requirement</i>). In
addition the absolute file path
 *     must be visible to all other machines that will potentially
access the file (<i>visibility
 *     requirement</i>).
 *
 *     <p>Whether data has hit non-volatile storage on the storage
nodes depends on the specific
 *     guarantees of the particular file system.
 *
 *     <p>The metadata updates to the file's parent directory are not
required to have reached
 *     a consistent state. It is permissible that some machines see
the file when listing the parent
 *     directory's contents while others do not, as long as access to
the file by its absolute path
 *     is possible on all nodes.</li>
 *
 *     <li>A <b>local file system</b> must support the POSIX
<i>close-to-open</i> semantics.
 *     Because the local file system does not have any fault tolerance
guarantees, no further
 *     requirements exist.
 *
 *     <p>The above implies specifically that data may still be in the
OS cache when considered
 *     persistent from the local file system's perspective. Crashes
that cause the OS cache to loose
 *     data are considered fatal to the local machine and are not
covered by the local file system's
 *     guarantees as defined by Flink.
 *
 *     <p>That means that computed results, checkpoints, and
savepoints that are written only to
 *     the local filesystem are not guaranteed to be recoverable from
the local machine's failure,
 *     making local file systems unsuitable for production setups.</li>
 * </ul>
 *
 * <h2>Updating File Contents</h2>
 *
 * Many file systems either do not support overwriting contents of
existing files at all, or do
 * not support consistent visibility of the updated contents in that
case. For that reason,
 * Flink's FileSystem does not support appending to existing files, or
seeking within output streams
 * so that previously written data could be overwritten.
 *
 * <h2>Overwriting Files</h2>
 *
 * Overwriting files is in general possible. A file is overwritten by
deleting it and creating
 * a new file. However, certain filesystems cannot make that change
synchronously visible
 * to all parties that have access to the file.
 * For example <a
href="https://aws.amazon.com/documentation/s3/">Amazon S3</a>
guarantees only
 * <i>eventual consistency</i> in the visibility of the file
replacement: Some machines may see
 * the old file, some machines may see the new file.
 *
 * <p>To avoid these consistency issues, t*he **implementations of
failure/recovery mechanisms in
 * Flink strictly avoid writing to the same file path more than once.*
 *
 * <h2>Thread Safety</h2>
 *
 * Implementations of {@code FileSystem} must be thread-safe: The same
instance of FileSystem
 * is frequently shared across multiple threads in Flink and must be
able to concurrently
 * create input/output streams and list file metadata.
 *
 * <p>The {@link FSDataOutputStream} and {@link FSDataOutputStream}
implementations are strictly
 * <b>not thread-safe</b>. Instances of the streams should also not be
passed between threads
 * in between read or write operations, because there are no
guarantees about the visibility of
 * operations across threads (many operations do not create memory fences).

On Thu, Feb 23, 2017 at 9:31 PM, Robert Metzger <rm...@apache.org> wrote:

> Hi Shai,
>
> I think we don't have so many users running Flink on Azure. Maybe you are
> the first to put some heavy load onto that infrastructure using Flink.
> I would guess that your problems are caused by the same root cause, just
> the way the job is being cancelled is a bit different based on what is
> happening.
> For the "The specified blob does not exist". Are there config options for
> the "NativeAzureFileSystem" that allow you to enable retries?
> Maybe its an "eventual consistency" issue of the underlying file store.
> (Flink is creating a directory against endpointA, and endpointB has not
> synced yet?).
>
> Can you check for this checkpoint the details page:
> 763             241/241         11:12:50        11:28:38
>       15m 48s
>
> It contains a breakdown what took so long.
> Was it the alignment time? the async time for the state to be snapshotted?
>
> If you want, you can also contact me privately and we do a Hangout session
> to look at the Flink UI together.
>
> Regards,
> Robert
>
>
>
>
>
> On Thu, Feb 23, 2017 at 8:59 AM, Shai Kaplan <Sh...@microsoft.com>
> wrote:
>
>> And now it's happening again
>>
>> -----Original Message-----
>> From: Shai Kaplan [mailto:Shai.Kaplan@microsoft.com]
>> Sent: Wednesday, February 22, 2017 12:02 PM
>> To: user@flink.apache.org
>> Subject: RE: Flink checkpointing gets stuck
>>
>> I changed the checkpoint interval to 30 minutes, and also switched
>> RocksDB predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay.
>> The problem hasn't exactly occurred since yesterday, but perhaps it just
>> takes it longer to happen again because the checkpoints are much less
>> frequent now.
>> I'm now pretty sure that the " Checkpoint Coordinator is suspending" is
>> not related to the other issue, because now I had a checkpoint that failed
>> because of that but then the next ones succeeded. I apologize for diverging
>> from the topic, but I think it's worth mentioning too, the exception was
>> "Could not materialize checkpoint" caused by "Could not flush and close the
>> file system output stream" caused by "com.microsoft.azure.storage.StorageException:
>> The specified blob does not exist". What could cause that? A temporary
>> failure in I/O with Azure blobs?
>>
>> Anyway, back to the topic, I didn't have checkpoints timing out, but I
>> did have one checkpoint that took significantly longer. This is what the
>> checkpoints history look like right now:
>>
>> ID      Status  Acknowledged    Trigger Time    Latest Acknowledgement
>> End to End Duration     State Size      Buffered During Alignment
>> 764             241/241         11:42:50        11:43:56
>>       1m 6s                   3.15 GB         30.4 MB
>> 763             241/241         11:12:50        11:28:38
>>       15m 48s                 3.11 GB         13.4 MB
>> 762             241/241         10:42:50        10:43:57
>>       1m 7s                   3.08 GB         12.8 MB
>> 761             241/241         10:12:50        10:13:58
>>       1m 8s                   3.03 GB         5.52 MB
>> 760             241/241         9:42:50         9:43:55
>>        1m 5s                   2.99 GB         607 KB
>> 759             241/241         9:12:50         9:13:52
>>        1m 1s                   2.94 GB         0 B
>> 758             241/241         8:42:50         8:43:46
>>        56s                     2.90 GB         5.55 MB
>> 757             241/241         8:12:50         8:14:20
>>        1m 30s                  2.85 GB         0 B
>> 756             121/241         7:41:49         7:42:09
>>        22s                     280 MB  0 B
>> 755             241/241         7:11:49         7:12:44
>>        55s                     2.81 GB         30 B
>>
>> The status wasn't copied well, but they all succeeded except for #756,
>> which failed for the reason I mentioned above.
>> As you can see, checkpoint #763 took a lot longer for no apparent reason,
>> so I'm guessing it's the same thing that caused the checkpoints to time out
>> at 30 minutes when they were saved every 10 seconds, only now it's less
>> severe because the load is much lower.
>> Any thoughts on what could cause that?
>>
>> -----Original Message-----
>> From: Ufuk Celebi [mailto:uce@apache.org]
>> Sent: Tuesday, February 21, 2017 4:54 PM
>> To: user@flink.apache.org
>> Subject: Re: Flink checkpointing gets stuck
>>
>> Hey Shai!
>>
>> Thanks for reporting this.
>>
>> It's hard to tell what causes this from your email, but could you check
>> the checkpoint interface
>> (https://na01.safelinks.protection.outlook.com/?url=https%
>> 3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-
>> release-1.3%2Fmonitoring%2Fcheckpoint_monitoring.html&data=
>> 02%7C01%7CShai.Kaplan%40microsoft.com%7C1cdb8bde8ee843676a00
>> 08d45a6984b7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
>> 6232856631405599&sdata=Ukf1%2Bahi00NVb3QVrTqUggYOSldheqvgMkJ
>> LsCzUWVc%3D&reserved=0)
>> and report how much progress the checkpoints make before timing out?
>>
>> The "Checkpoint Coordinator is suspending" message indicates that the job
>> failed and the checkpoint coordinator is shut down because of that. Can you
>> check the TaskManager and JobManager logs if other errors are reported?
>> Feel free to share them. Then I could help with going over them.
>>
>> – Ufuk
>>
>>
>> On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com>
>> wrote:
>> > Hi.
>> >
>> > I'm running a Flink 1.2 job with a 10 seconds checkpoint interval.
>> > After some running time (minutes-hours) Flink fails to save
>> > checkpoints, and stops processing records (I'm not sure if the
>> > checkpointing failure is the cause of the problem or just a symptom).
>> >
>> > After several checkpoints that take some seconds each, they start
>> > failing due to 30 minutes timeout.
>> >
>> > When I restart one of the Task Manager services (just to get the job
>> > restarted), the job is recovered from the last successful checkpoint
>> > (the state size continues to grow, so it's probably not the reason for
>> > the failure), advances somewhat, saves some more checkpoints, and then
>> > enters the failing state again.
>> >
>> > One of the times it happened, the first failed checkpoint failed due
>> > to "Checkpoint Coordinator is suspending.", so it might be an
>> > indicator for the cause of the problem, but looking into Flink's code
>> > I can't see how a running job could get to this state.
>> >
>> > I am using RocksDB for state, and the state is saved to Azure Blob
>> > Store, using the NativeAzureFileSystem HDFS connector over the wasbs
>> protocol.
>> >
>> > Any ideas? Possibly a bug in Flink or RocksDB?
>>
>
>

Re: Flink checkpointing gets stuck

Posted by Robert Metzger <rm...@apache.org>.

Hi Shai,

I think we don't have so many users running Flink on Azure. Maybe you are
the first to put some heavy load onto that infrastructure using Flink.
I would guess that your problems are caused by the same root cause, just
the way the job is being cancelled is a bit different based on what is
happening.
For the "The specified blob does not exist". Are there config options for
the "NativeAzureFileSystem" that allow you to enable retries?
Maybe its an "eventual consistency" issue of the underlying file store.
(Flink is creating a directory against endpointA, and endpointB has not
synced yet?).

Can you check for this checkpoint the details page:
763             241/241         11:12:50        11:28:38
    15m 48s

It contains a breakdown what took so long.
Was it the alignment time? the async time for the state to be snapshotted?

If you want, you can also contact me privately and we do a Hangout session
to look at the Flink UI together.

Regards,
Robert





On Thu, Feb 23, 2017 at 8:59 AM, Shai Kaplan <Sh...@microsoft.com>
wrote:

> And now it's happening again
>
> -----Original Message-----
> From: Shai Kaplan [mailto:Shai.Kaplan@microsoft.com]
> Sent: Wednesday, February 22, 2017 12:02 PM
> To: user@flink.apache.org
> Subject: RE: Flink checkpointing gets stuck
>
> I changed the checkpoint interval to 30 minutes, and also switched RocksDB
> predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay. The
> problem hasn't exactly occurred since yesterday, but perhaps it just takes
> it longer to happen again because the checkpoints are much less frequent
> now.
> I'm now pretty sure that the " Checkpoint Coordinator is suspending" is
> not related to the other issue, because now I had a checkpoint that failed
> because of that but then the next ones succeeded. I apologize for diverging
> from the topic, but I think it's worth mentioning too, the exception was
> "Could not materialize checkpoint" caused by "Could not flush and close the
> file system output stream" caused by "com.microsoft.azure.storage.StorageException:
> The specified blob does not exist". What could cause that? A temporary
> failure in I/O with Azure blobs?
>
> Anyway, back to the topic, I didn't have checkpoints timing out, but I did
> have one checkpoint that took significantly longer. This is what the
> checkpoints history look like right now:
>
> ID      Status  Acknowledged    Trigger Time    Latest Acknowledgement
> End to End Duration     State Size      Buffered During Alignment
> 764             241/241         11:42:50        11:43:56
>       1m 6s                   3.15 GB         30.4 MB
> 763             241/241         11:12:50        11:28:38
>       15m 48s                 3.11 GB         13.4 MB
> 762             241/241         10:42:50        10:43:57
>       1m 7s                   3.08 GB         12.8 MB
> 761             241/241         10:12:50        10:13:58
>       1m 8s                   3.03 GB         5.52 MB
> 760             241/241         9:42:50         9:43:55
>      1m 5s                   2.99 GB         607 KB
> 759             241/241         9:12:50         9:13:52
>      1m 1s                   2.94 GB         0 B
> 758             241/241         8:42:50         8:43:46
>      56s                     2.90 GB         5.55 MB
> 757             241/241         8:12:50         8:14:20
>      1m 30s                  2.85 GB         0 B
> 756             121/241         7:41:49         7:42:09
>      22s                     280 MB  0 B
> 755             241/241         7:11:49         7:12:44
>      55s                     2.81 GB         30 B
>
> The status wasn't copied well, but they all succeeded except for #756,
> which failed for the reason I mentioned above.
> As you can see, checkpoint #763 took a lot longer for no apparent reason,
> so I'm guessing it's the same thing that caused the checkpoints to time out
> at 30 minutes when they were saved every 10 seconds, only now it's less
> severe because the load is much lower.
> Any thoughts on what could cause that?
>
> -----Original Message-----
> From: Ufuk Celebi [mailto:uce@apache.org]
> Sent: Tuesday, February 21, 2017 4:54 PM
> To: user@flink.apache.org
> Subject: Re: Flink checkpointing gets stuck
>
> Hey Shai!
>
> Thanks for reporting this.
>
> It's hard to tell what causes this from your email, but could you check
> the checkpoint interface
> (https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-
> docs-release-1.3%2Fmonitoring%2Fcheckpoint_monitoring.html&
> data=02%7C01%7CShai.Kaplan%40microsoft.com%7C1cdb8bde8ee843676a0008d45a69
> 84b7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%
> 7C636232856631405599&sdata=Ukf1%2Bahi00NVb3QVrTqUggYOSldheqvgM
> kJLsCzUWVc%3D&reserved=0)
> and report how much progress the checkpoints make before timing out?
>
> The "Checkpoint Coordinator is suspending" message indicates that the job
> failed and the checkpoint coordinator is shut down because of that. Can you
> check the TaskManager and JobManager logs if other errors are reported?
> Feel free to share them. Then I could help with going over them.
>
> – Ufuk
>
>
> On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com>
> wrote:
> > Hi.
> >
> > I'm running a Flink 1.2 job with a 10 seconds checkpoint interval.
> > After some running time (minutes-hours) Flink fails to save
> > checkpoints, and stops processing records (I'm not sure if the
> > checkpointing failure is the cause of the problem or just a symptom).
> >
> > After several checkpoints that take some seconds each, they start
> > failing due to 30 minutes timeout.
> >
> > When I restart one of the Task Manager services (just to get the job
> > restarted), the job is recovered from the last successful checkpoint
> > (the state size continues to grow, so it's probably not the reason for
> > the failure), advances somewhat, saves some more checkpoints, and then
> > enters the failing state again.
> >
> > One of the times it happened, the first failed checkpoint failed due
> > to "Checkpoint Coordinator is suspending.", so it might be an
> > indicator for the cause of the problem, but looking into Flink's code
> > I can't see how a running job could get to this state.
> >
> > I am using RocksDB for state, and the state is saved to Azure Blob
> > Store, using the NativeAzureFileSystem HDFS connector over the wasbs
> protocol.
> >
> > Any ideas? Possibly a bug in Flink or RocksDB?
>

RE: Flink checkpointing gets stuck

Posted by Shai Kaplan <Sh...@microsoft.com>.

And now it's happening again

-----Original Message-----
From: Shai Kaplan [mailto:Shai.Kaplan@microsoft.com] 
Sent: Wednesday, February 22, 2017 12:02 PM
To: user@flink.apache.org
Subject: RE: Flink checkpointing gets stuck

I changed the checkpoint interval to 30 minutes, and also switched RocksDB predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay. The problem hasn't exactly occurred since yesterday, but perhaps it just takes it longer to happen again because the checkpoints are much less frequent now.
I'm now pretty sure that the " Checkpoint Coordinator is suspending" is not related to the other issue, because now I had a checkpoint that failed because of that but then the next ones succeeded. I apologize for diverging from the topic, but I think it's worth mentioning too, the exception was "Could not materialize checkpoint" caused by "Could not flush and close the file system output stream" caused by "com.microsoft.azure.storage.StorageException: The specified blob does not exist". What could cause that? A temporary failure in I/O with Azure blobs?

Anyway, back to the topic, I didn't have checkpoints timing out, but I did have one checkpoint that took significantly longer. This is what the checkpoints history look like right now:

ID 	Status 	Acknowledged 	Trigger Time 	Latest Acknowledgement 	End to End Duration 	State Size 	Buffered During Alignment 	
764 		241/241 	11:42:50 	11:43:56 			1m 6s 			3.15 GB 	30.4 MB 	
763 		241/241 	11:12:50 	11:28:38 			15m 48s 		3.11 GB 	13.4 MB 	
762 		241/241 	10:42:50 	10:43:57 			1m 7s 			3.08 GB 	12.8 MB 	
761 		241/241 	10:12:50 	10:13:58 			1m 8s 			3.03 GB 	5.52 MB 	
760 		241/241 	9:42:50 	9:43:55 			1m 5s 			2.99 GB 	607 KB 	
759 		241/241 	9:12:50 	9:13:52 			1m 1s 			2.94 GB 	0 B 	
758 		241/241 	8:42:50 	8:43:46 			56s 			2.90 GB 	5.55 MB 	
757 		241/241 	8:12:50 	8:14:20 			1m 30s 			2.85 GB 	0 B 	
756 		121/241 	7:41:49 	7:42:09 			22s 			280 MB 	0 B 	
755 		241/241 	7:11:49 	7:12:44 			55s 			2.81 GB 	30 B

The status wasn't copied well, but they all succeeded except for #756, which failed for the reason I mentioned above.
As you can see, checkpoint #763 took a lot longer for no apparent reason, so I'm guessing it's the same thing that caused the checkpoints to time out at 30 minutes when they were saved every 10 seconds, only now it's less severe because the load is much lower.
Any thoughts on what could cause that?

-----Original Message-----
From: Ufuk Celebi [mailto:uce@apache.org]
Sent: Tuesday, February 21, 2017 4:54 PM
To: user@flink.apache.org
Subject: Re: Flink checkpointing gets stuck

Hey Shai!

Thanks for reporting this.

It's hard to tell what causes this from your email, but could you check the checkpoint interface
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fmonitoring%2Fcheckpoint_monitoring.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C1cdb8bde8ee843676a0008d45a6984b7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232856631405599&sdata=Ukf1%2Bahi00NVb3QVrTqUggYOSldheqvgMkJLsCzUWVc%3D&reserved=0)
and report how much progress the checkpoints make before timing out?

The "Checkpoint Coordinator is suspending" message indicates that the job failed and the checkpoint coordinator is shut down because of that. Can you check the TaskManager and JobManager logs if other errors are reported? Feel free to share them. Then I could help with going over them.

– Ufuk

On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com> wrote:
> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. 
> After some running time (minutes-hours) Flink fails to save 
> checkpoints, and stops processing records (I'm not sure if the 
> checkpointing failure is the cause of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start 
> failing due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job 
> restarted), the job is recovered from the last successful checkpoint 
> (the state size continues to grow, so it's probably not the reason for 
> the failure), advances somewhat, saves some more checkpoints, and then 
> enters the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due 
> to "Checkpoint Coordinator is suspending.", so it might be an 
> indicator for the cause of the problem, but looking into Flink's code 
> I can't see how a running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob 
> Store, using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?

RE: Flink checkpointing gets stuck

Posted by Shai Kaplan <Sh...@microsoft.com>.

I changed the checkpoint interval to 30 minutes, and also switched RocksDB predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay. The problem hasn't exactly occurred since yesterday, but perhaps it just takes it longer to happen again because the checkpoints are much less frequent now.
I'm now pretty sure that the " Checkpoint Coordinator is suspending" is not related to the other issue, because now I had a checkpoint that failed because of that but then the next ones succeeded. I apologize for diverging from the topic, but I think it's worth mentioning too, the exception was "Could not materialize checkpoint" caused by "Could not flush and close the file system output stream" caused by "com.microsoft.azure.storage.StorageException: The specified blob does not exist". What could cause that? A temporary failure in I/O with Azure blobs?

Anyway, back to the topic, I didn't have checkpoints timing out, but I did have one checkpoint that took significantly longer. This is what the checkpoints history look like right now:

ID 	Status 	Acknowledged 	Trigger Time 	Latest Acknowledgement 	End to End Duration 	State Size 	Buffered During Alignment 	
764 		241/241 	11:42:50 	11:43:56 			1m 6s 			3.15 GB 	30.4 MB 	
763 		241/241 	11:12:50 	11:28:38 			15m 48s 		3.11 GB 	13.4 MB 	
762 		241/241 	10:42:50 	10:43:57 			1m 7s 			3.08 GB 	12.8 MB 	
761 		241/241 	10:12:50 	10:13:58 			1m 8s 			3.03 GB 	5.52 MB 	
760 		241/241 	9:42:50 	9:43:55 			1m 5s 			2.99 GB 	607 KB 	
759 		241/241 	9:12:50 	9:13:52 			1m 1s 			2.94 GB 	0 B 	
758 		241/241 	8:42:50 	8:43:46 			56s 			2.90 GB 	5.55 MB 	
757 		241/241 	8:12:50 	8:14:20 			1m 30s 			2.85 GB 	0 B 	
756 		121/241 	7:41:49 	7:42:09 			22s 			280 MB 	0 B 	
755 		241/241 	7:11:49 	7:12:44 			55s 			2.81 GB 	30 B

The status wasn't copied well, but they all succeeded except for #756, which failed for the reason I mentioned above.
As you can see, checkpoint #763 took a lot longer for no apparent reason, so I'm guessing it's the same thing that caused the checkpoints to time out at 30 minutes when they were saved every 10 seconds, only now it's less severe because the load is much lower.
Any thoughts on what could cause that?

-----Original Message-----
From: Ufuk Celebi [mailto:uce@apache.org] 
Sent: Tuesday, February 21, 2017 4:54 PM
To: user@flink.apache.org
Subject: Re: Flink checkpointing gets stuck

Hey Shai!

Thanks for reporting this.

It's hard to tell what causes this from your email, but could you check the checkpoint interface
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fmonitoring%2Fcheckpoint_monitoring.html&data=02%7C01%7CShai.Kaplan%40microsoft.com%7C1cdb8bde8ee843676a0008d45a6984b7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636232856631405599&sdata=Ukf1%2Bahi00NVb3QVrTqUggYOSldheqvgMkJLsCzUWVc%3D&reserved=0)
and report how much progress the checkpoints make before timing out?

The "Checkpoint Coordinator is suspending" message indicates that the job failed and the checkpoint coordinator is shut down because of that. Can you check the TaskManager and JobManager logs if other errors are reported? Feel free to share them. Then I could help with going over them.

– Ufuk

On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com> wrote:
> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. 
> After some running time (minutes-hours) Flink fails to save 
> checkpoints, and stops processing records (I'm not sure if the 
> checkpointing failure is the cause of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start 
> failing due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job 
> restarted), the job is recovered from the last successful checkpoint 
> (the state size continues to grow, so it's probably not the reason for 
> the failure), advances somewhat, saves some more checkpoints, and then 
> enters the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due 
> to "Checkpoint Coordinator is suspending.", so it might be an 
> indicator for the cause of the problem, but looking into Flink's code 
> I can't see how a running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob 
> Store, using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?

Re: Flink checkpointing gets stuck

Posted by Ufuk Celebi <uc...@apache.org>.

Hey Shai!

Thanks for reporting this.

It's hard to tell what causes this from your email, but could you
check the checkpoint interface
(https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/checkpoint_monitoring.html)
and report how much progress the checkpoints make before timing out?

The "Checkpoint Coordinator is suspending" message indicates that the
job failed and the checkpoint coordinator is shut down because of
that. Can you check the TaskManager and JobManager logs if other
errors are reported? Feel free to share them. Then I could help with
going over them.

– Ufuk

On Tue, Feb 21, 2017 at 2:47 PM, Shai Kaplan <Sh...@microsoft.com> wrote:
> Hi.
>
> I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After
> some running time (minutes-hours) Flink fails to save checkpoints, and stops
> processing records (I'm not sure if the checkpointing failure is the cause
> of the problem or just a symptom).
>
> After several checkpoints that take some seconds each, they start failing
> due to 30 minutes timeout.
>
> When I restart one of the Task Manager services (just to get the job
> restarted), the job is recovered from the last successful checkpoint (the
> state size continues to grow, so it's probably not the reason for the
> failure), advances somewhat, saves some more checkpoints, and then enters
> the failing state again.
>
> One of the times it happened, the first failed checkpoint failed due to
> "Checkpoint Coordinator is suspending.", so it might be an indicator for the
> cause of the problem, but looking into Flink's code I can't see how a
> running job could get to this state.
>
> I am using RocksDB for state, and the state is saved to Azure Blob Store,
> using the NativeAzureFileSystem HDFS connector over the wasbs protocol.
>
> Any ideas? Possibly a bug in Flink or RocksDB?