You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ufuk Celebi <u....@fu-berlin.de> on 2015/02/02 18:45:31 UTC
Task manager memory configuration with intermediate results
Currently, the memory configuration of a task manager encompasses two things:
1) NETWORK buffers: Fixed amount of memory for the network buffer pool (default: 2048 buffers, each 32 KB => 64 MB)
2) OPERATOR buffers: A configurable fraction of the available heap memory (default 0.7) for the memory manager used by internal operators like sort or hash
With the recently added supported for intermediate results, intermediate results live in the network stack and use buffers from the network buffer pool. Currently, our memory management is not a problem, because we only support ephemeral intermediate results, which are directly consumed in a pipelined fashion (i.e. the buffer pools are short-lived).
But with the upcoming support for persistent intermediate results (https://github.com/apache/flink/pull/356) and fine-grained fault tolerance, we need to rethink how we configure/divide the available memory between the network stack and the operators as the network buffer pools will live longer.
I would suggest the following:
1) Remove the configuration for network buffers
2) Keep the fraction configuration, but internally divide it between network stack and operators in a 50:50 fashion (in the future with dynamic memory management, we will not even need to statically divide the memory between network stack and operators).
What do you think? Is this reasonable? Should the division be configurable as well?
– Ufuk
Re: Task manager memory configuration with intermediate results
Posted by Max Michels <ma...@data-artisans.com>.
+1 The static memory assignment of the network buffer tool caused some
problems for users in the past. Ultimately, dynamic memory management
would be desirable. Until then, let's remove the absolute value
configuration for the network buffers and introduce a parameter to
divide the heap memory among network and non-network code.
On Tue, Feb 3, 2015 at 10:48 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Yes, I would really like to get rid of the distinction between operator and
> network buffers.
> Having all buffers been taken from the same pool is a good step towards
> that goal. Until the assignment is dynamic, I prefer to have a config
> option for the network / operator ratio.
>
> +1 for the proposal and a ratio config option.
>
> 2015-02-03 9:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> I like this approach and would suggest to make the ratio configurable. The
>> default could be 50/50 or 60/40 (op heap / net heap)
>>
>> On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u....@fu-berlin.de> wrote:
>>
>> > Currently, the memory configuration of a task manager encompasses two
>> > things:
>> >
>> > 1) NETWORK buffers: Fixed amount of memory for the network buffer pool
>> > (default: 2048 buffers, each 32 KB => 64 MB)
>> >
>> > 2) OPERATOR buffers: A configurable fraction of the available heap memory
>> > (default 0.7) for the memory manager used by internal operators like sort
>> > or hash
>> >
>> > With the recently added supported for intermediate results, intermediate
>> > results live in the network stack and use buffers from the network buffer
>> > pool. Currently, our memory management is not a problem, because we only
>> > support ephemeral intermediate results, which are directly consumed in a
>> > pipelined fashion (i.e. the buffer pools are short-lived).
>> >
>> > But with the upcoming support for persistent intermediate results (
>> > https://github.com/apache/flink/pull/356) and fine-grained fault
>> > tolerance, we need to rethink how we configure/divide the available
>> memory
>> > between the network stack and the operators as the network buffer pools
>> > will live longer.
>> >
>> > I would suggest the following:
>> >
>> > 1) Remove the configuration for network buffers
>> >
>> > 2) Keep the fraction configuration, but internally divide it between
>> > network stack and operators in a 50:50 fashion (in the future with
>> dynamic
>> > memory management, we will not even need to statically divide the memory
>> > between network stack and operators).
>> >
>> >
>> > What do you think? Is this reasonable? Should the division be
>> configurable
>> > as well?
>> >
>> > – Ufuk
>>
Re: Task manager memory configuration with intermediate results
Posted by Fabian Hueske <fh...@gmail.com>.
Yes, I would really like to get rid of the distinction between operator and
network buffers.
Having all buffers been taken from the same pool is a good step towards
that goal. Until the assignment is dynamic, I prefer to have a config
option for the network / operator ratio.
+1 for the proposal and a ratio config option.
2015-02-03 9:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
> I like this approach and would suggest to make the ratio configurable. The
> default could be 50/50 or 60/40 (op heap / net heap)
>
> On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u....@fu-berlin.de> wrote:
>
> > Currently, the memory configuration of a task manager encompasses two
> > things:
> >
> > 1) NETWORK buffers: Fixed amount of memory for the network buffer pool
> > (default: 2048 buffers, each 32 KB => 64 MB)
> >
> > 2) OPERATOR buffers: A configurable fraction of the available heap memory
> > (default 0.7) for the memory manager used by internal operators like sort
> > or hash
> >
> > With the recently added supported for intermediate results, intermediate
> > results live in the network stack and use buffers from the network buffer
> > pool. Currently, our memory management is not a problem, because we only
> > support ephemeral intermediate results, which are directly consumed in a
> > pipelined fashion (i.e. the buffer pools are short-lived).
> >
> > But with the upcoming support for persistent intermediate results (
> > https://github.com/apache/flink/pull/356) and fine-grained fault
> > tolerance, we need to rethink how we configure/divide the available
> memory
> > between the network stack and the operators as the network buffer pools
> > will live longer.
> >
> > I would suggest the following:
> >
> > 1) Remove the configuration for network buffers
> >
> > 2) Keep the fraction configuration, but internally divide it between
> > network stack and operators in a 50:50 fashion (in the future with
> dynamic
> > memory management, we will not even need to statically divide the memory
> > between network stack and operators).
> >
> >
> > What do you think? Is this reasonable? Should the division be
> configurable
> > as well?
> >
> > – Ufuk
>
Re: Task manager memory configuration with intermediate results
Posted by Stephan Ewen <se...@apache.org>.
I like this approach and would suggest to make the ratio configurable. The
default could be 50/50 or 60/40 (op heap / net heap)
On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u....@fu-berlin.de> wrote:
> Currently, the memory configuration of a task manager encompasses two
> things:
>
> 1) NETWORK buffers: Fixed amount of memory for the network buffer pool
> (default: 2048 buffers, each 32 KB => 64 MB)
>
> 2) OPERATOR buffers: A configurable fraction of the available heap memory
> (default 0.7) for the memory manager used by internal operators like sort
> or hash
>
> With the recently added supported for intermediate results, intermediate
> results live in the network stack and use buffers from the network buffer
> pool. Currently, our memory management is not a problem, because we only
> support ephemeral intermediate results, which are directly consumed in a
> pipelined fashion (i.e. the buffer pools are short-lived).
>
> But with the upcoming support for persistent intermediate results (
> https://github.com/apache/flink/pull/356) and fine-grained fault
> tolerance, we need to rethink how we configure/divide the available memory
> between the network stack and the operators as the network buffer pools
> will live longer.
>
> I would suggest the following:
>
> 1) Remove the configuration for network buffers
>
> 2) Keep the fraction configuration, but internally divide it between
> network stack and operators in a 50:50 fashion (in the future with dynamic
> memory management, we will not even need to statically divide the memory
> between network stack and operators).
>
>
> What do you think? Is this reasonable? Should the division be configurable
> as well?
>
> – Ufuk