Flink job backpressure and TM network buffer memory

[Yuval Itzchakov <yu...@gmail.com>]

Hi,
Flink 1.14.2

I have a job which is highly backpressured. I am trying to get down to the
root cause. When looking at the backpressured operator flamegraph, I see it
is waiting on requesting buffers in order to write records to downstream
operators:

[image: image.png]

However, when looking at the TM configuration, I see that network memory
usage is relatively low. I might be misunderstanding the metric there, but
I would expect it to be much more utilized given that a lot of the subtasks
are waiting for output buffers to write data:

[image: image.png]
[image: image.png]

On the other hand, the node causing the backpressure isn't really always
100% utilized:
[image: image.png]
[image: image.png]

What am I missing?

-- 
Best Regards,
Yuval Itzchakov.

Re: Flink job backpressure and TM network buffer memory

[David Morávek <dm...@apache.org>]

Hi Yuval,

Few notes:

- For the operator that's causing backpressure (the first one that is not
marked as backpressured), I think you should be looking for the on-cpu
graph instead. I'd suspect a slow per-element code path there. (for example
it can be caused by state access in case you're using rocksdb)
- Being blocked on the "request memory" segment IMO means that the output
buffer being depleted, in other words, the next operator is not catching up
with the input. (please not that I'm not an expert on the low-level runtime
code paths).

I hope this helps, I'm not really able to make any conclusion based on the
available information.

Best,
D.

On Tue, Dec 28, 2021 at 2:41 PM Yuval Itzchakov <yu...@gmail.com> wrote:

> Hi,
> Flink 1.14.2
>
> I have a job which is highly backpressured. I am trying to get down to the
> root cause. When looking at the backpressured operator flamegraph, I see it
> is waiting on requesting buffers in order to write records to downstream
> operators:
>
> [image: image.png]
>
> However, when looking at the TM configuration, I see that network memory
> usage is relatively low. I might be misunderstanding the metric there, but
> I would expect it to be much more utilized given that a lot of the subtasks
> are waiting for output buffers to write data:
>
> [image: image.png]
> [image: image.png]
>
> On the other hand, the node causing the backpressure isn't really always
> 100% utilized:
> [image: image.png]
> [image: image.png]
>
> What am I missing?
>
> --
> Best Regards,
> Yuval Itzchakov.
>