You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Rex Fenley <Re...@remind101.com> on 2020/12/15 07:36:34 UTC
Direct Memory full
Hello,
Our job consistently shows
Outside JVM
Type
Count
Used
Capacity
*Direct* 32,839 1.03 GB 1.03 GB
for direct memory.
Is it typical for it to be full? What are the consequences that we may not
be noticing of direct memory being full?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Direct Memory full
Posted by Rex Fenley <Re...@remind101.com>.
Ok, thanks. I've said this in another thread but everything seems to go
completely idle during checkpoints while waiting on 1 operator, there's no
CPU usage, hardly any disk usage. I'll assume it's something else then.
On Wed, Dec 16, 2020 at 10:42 AM Robert Metzger <rm...@apache.org> wrote:
> I don't think the direct memory is causing any performance bottlenecks.
> The backpressure is probably caused by something else (high CPU load, slow
> external system, data skew)
>
> On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <st...@gmail.com> wrote:
>
>> if you are running out of direct buffer, you will see "java.lang.OutOfMemoryError:
>> Direct buffer memory"
>>
>> On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <Re...@remind101.com> wrote:
>>
>>> Thanks for the reply. If what I'm understanding is correct there's no
>>> chance of an OOM, but since direct memory is for I/O, it being completely
>>> filled may be a sign of backpressure? Currently one of our operators takes
>>> a tremendous amount of time to align during a checkpoint. Could increasing
>>> direct memory help checkpointing by improving I/O performance across the
>>> whole plan (assuming I/O is at least part of the bottleneck)?
>>>
>>> On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rm...@apache.org>
>>> wrote:
>>>
>>>> Hey Rex,
>>>>
>>>> the direct memory is used for IO. There is no concept of direct memory
>>>> being "full". The only thing that can happen is that you have something in
>>>> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
>>>> process, and you run out of your memory allowance. The direct memory is
>>>> allocated outside of the heap's upper limit, thus you could run out of the
>>>> budget.
>>>> But Flink is usually properly configuring the memory limits correctly,
>>>> to avoid running into those situations.
>>>>
>>>> tl;dr: you don't need to worry about this.
>>>>
>>>>
>>>> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <Re...@remind101.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Our job consistently shows
>>>>> Outside JVM
>>>>> Type
>>>>> Count
>>>>> Used
>>>>> Capacity
>>>>> *Direct* 32,839 1.03 GB 1.03 GB
>>>>> for direct memory.
>>>>>
>>>>> Is it typical for it to be full? What are the consequences that we may
>>>>> not be noticing of direct memory being full?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>>
>>>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>>>
>>>>>
>>>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>>>> <https://www.facebook.com/remindhq>
>>>>>
>>>>
>>>
>>> --
>>>
>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>
>>>
>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>> <https://www.facebook.com/remindhq>
>>>
>>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Direct Memory full
Posted by Robert Metzger <rm...@apache.org>.
I don't think the direct memory is causing any performance bottlenecks. The
backpressure is probably caused by something else (high CPU load, slow
external system, data skew)
On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <st...@gmail.com> wrote:
> if you are running out of direct buffer, you will see "java.lang.OutOfMemoryError:
> Direct buffer memory"
>
> On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <Re...@remind101.com> wrote:
>
>> Thanks for the reply. If what I'm understanding is correct there's no
>> chance of an OOM, but since direct memory is for I/O, it being completely
>> filled may be a sign of backpressure? Currently one of our operators takes
>> a tremendous amount of time to align during a checkpoint. Could increasing
>> direct memory help checkpointing by improving I/O performance across the
>> whole plan (assuming I/O is at least part of the bottleneck)?
>>
>> On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rm...@apache.org>
>> wrote:
>>
>>> Hey Rex,
>>>
>>> the direct memory is used for IO. There is no concept of direct memory
>>> being "full". The only thing that can happen is that you have something in
>>> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
>>> process, and you run out of your memory allowance. The direct memory is
>>> allocated outside of the heap's upper limit, thus you could run out of the
>>> budget.
>>> But Flink is usually properly configuring the memory limits correctly,
>>> to avoid running into those situations.
>>>
>>> tl;dr: you don't need to worry about this.
>>>
>>>
>>> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <Re...@remind101.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Our job consistently shows
>>>> Outside JVM
>>>> Type
>>>> Count
>>>> Used
>>>> Capacity
>>>> *Direct* 32,839 1.03 GB 1.03 GB
>>>> for direct memory.
>>>>
>>>> Is it typical for it to be full? What are the consequences that we may
>>>> not be noticing of direct memory being full?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>>
>>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>>
>>>>
>>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>>> <https://www.facebook.com/remindhq>
>>>>
>>>
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
Re: Direct Memory full
Posted by Steven Wu <st...@gmail.com>.
if you are running out of direct buffer, you will see
"java.lang.OutOfMemoryError:
Direct buffer memory"
On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <Re...@remind101.com> wrote:
> Thanks for the reply. If what I'm understanding is correct there's no
> chance of an OOM, but since direct memory is for I/O, it being completely
> filled may be a sign of backpressure? Currently one of our operators takes
> a tremendous amount of time to align during a checkpoint. Could increasing
> direct memory help checkpointing by improving I/O performance across the
> whole plan (assuming I/O is at least part of the bottleneck)?
>
> On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rm...@apache.org>
> wrote:
>
>> Hey Rex,
>>
>> the direct memory is used for IO. There is no concept of direct memory
>> being "full". The only thing that can happen is that you have something in
>> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
>> process, and you run out of your memory allowance. The direct memory is
>> allocated outside of the heap's upper limit, thus you could run out of the
>> budget.
>> But Flink is usually properly configuring the memory limits correctly, to
>> avoid running into those situations.
>>
>> tl;dr: you don't need to worry about this.
>>
>>
>> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <Re...@remind101.com> wrote:
>>
>>> Hello,
>>>
>>> Our job consistently shows
>>> Outside JVM
>>> Type
>>> Count
>>> Used
>>> Capacity
>>> *Direct* 32,839 1.03 GB 1.03 GB
>>> for direct memory.
>>>
>>> Is it typical for it to be full? What are the consequences that we may
>>> not be noticing of direct memory being full?
>>>
>>> Thanks!
>>>
>>> --
>>>
>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>
>>>
>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>> <https://www.facebook.com/remindhq>
>>>
>>
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
Re: Direct Memory full
Posted by Rex Fenley <Re...@remind101.com>.
Thanks for the reply. If what I'm understanding is correct there's no
chance of an OOM, but since direct memory is for I/O, it being completely
filled may be a sign of backpressure? Currently one of our operators takes
a tremendous amount of time to align during a checkpoint. Could increasing
direct memory help checkpointing by improving I/O performance across the
whole plan (assuming I/O is at least part of the bottleneck)?
On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rm...@apache.org> wrote:
> Hey Rex,
>
> the direct memory is used for IO. There is no concept of direct memory
> being "full". The only thing that can happen is that you have something in
> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
> process, and you run out of your memory allowance. The direct memory is
> allocated outside of the heap's upper limit, thus you could run out of the
> budget.
> But Flink is usually properly configuring the memory limits correctly, to
> avoid running into those situations.
>
> tl;dr: you don't need to worry about this.
>
>
> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <Re...@remind101.com> wrote:
>
>> Hello,
>>
>> Our job consistently shows
>> Outside JVM
>> Type
>> Count
>> Used
>> Capacity
>> *Direct* 32,839 1.03 GB 1.03 GB
>> for direct memory.
>>
>> Is it typical for it to be full? What are the consequences that we may
>> not be noticing of direct memory being full?
>>
>> Thanks!
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Direct Memory full
Posted by Robert Metzger <rm...@apache.org>.
Hey Rex,
the direct memory is used for IO. There is no concept of direct memory
being "full". The only thing that can happen is that you have something in
place (Kubernetes, YARN) that limits / enforces the memory use of a Flink
process, and you run out of your memory allowance. The direct memory is
allocated outside of the heap's upper limit, thus you could run out of the
budget.
But Flink is usually properly configuring the memory limits correctly, to
avoid running into those situations.
tl;dr: you don't need to worry about this.
On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <Re...@remind101.com> wrote:
> Hello,
>
> Our job consistently shows
> Outside JVM
> Type
> Count
> Used
> Capacity
> *Direct* 32,839 1.03 GB 1.03 GB
> for direct memory.
>
> Is it typical for it to be full? What are the consequences that we may not
> be noticing of direct memory being full?
>
> Thanks!
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>