You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vignesh Ramesh <vi...@gmail.com> on 2022/04/07 04:33:56 UTC
Produnction : Flink 1.14.4 : Kafka reader threads blocked
Hi Team,
We are using flink 1.14.4 and facing issues in production where our threads
are blocked waiting for
*LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346).*
*"Legacy Source Thread - Source: ReindexBatchProcess Source (61/64)#0"
Id=376 WAITING on java.util.concurrent.CompletableFuture$Signaller@6ddd0af
at sun.misc.Unsafe.park(Native Method) - waiting on
java.util.concurrent.CompletableFuture$Signaller@6ddd0af at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.flink.runtime.io
<http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346)
at org.apache.flink.runtime.io
<http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:318)*
Attaching the thread dump with this mail. We also don't see any issues in
downstream processing . The network buffer memory and heap memory is also
used very less please refer below pic.
[image: image.png]
Kindly let us know if we can do anything to solve this issue.. We are also
open for commercial support.
Re: Produnction : Flink 1.14.4 : Kafka reader threads blocked
Posted by yu'an huang <h....@gmail.com>.
Hi Vignesh,
I think you can check the following things:
1. Check the cpu usage of the workers. Are they close to zero or almost full?
2. Any back pressure happened in downstream tasks?
3. Is the fullGC significant serious?
Best,
Yuan
> On 7 Apr 2022, at 12:33 PM, Vignesh Ramesh <vi...@gmail.com> wrote:
>
> Hi Team,
>
> We are using flink 1.14.4 and facing issues in production where our threads are blocked waiting for LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346).
>
> "Legacy Source Thread - Source: ReindexBatchProcess Source (61/64)#0" Id=376 WAITING on java.util.concurrent.CompletableFuture$Signaller@6ddd0af
> at sun.misc.Unsafe.park(Native Method)
> - waiting on java.util.concurrent.CompletableFuture$Signaller@6ddd0af
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at org.apache.flink.runtime.io <http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:346)
> at org.apache.flink.runtime.io <http://org.apache.flink.runtime.io/>.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:318)
>
> Attaching the thread dump with this mail. We also don't see any issues in downstream processing . The network buffer memory and heap memory is also used very less please refer below pic.
>
> <image.png>
>
> Kindly let us know if we can do anything to solve this issue.. We are also open for commercial support.
> <threaddump-latest.txt>