You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by nragon <nu...@wedotechnologies.com> on 2017/04/06 14:36:59 UTC

Re: Should I decrease the taskmanager.memory.fraction ?

Hi,

Regarding this topic, does flink uses memory segments for window states?
Is memory managment only "used at is full capacity" for batch?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475p12540.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Should I decrease the taskmanager.memory.fraction ?

Posted by nragon <nu...@wedotechnologies.com>.
Ok, got it :)
But since memory segments are good to avoid gc problems, will streaming jobs
suffer from gc when keeping window or any other states?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475p12544.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Should I decrease the taskmanager.memory.fraction ?

Posted by Fabian Hueske <fh...@gmail.com>.
Yes, by default managed memory is lazily allocated. Since streaming
applications don't request management memory, it won't be allocated

2017-04-06 16:52 GMT+02:00 nragon <nu...@wedotechnologies.com>:

> Like stephan said we don't need to worry about memory managment with
> streaming? Meaning if I have 3 taskmanagers with 2Gb each, memory segments
> won't be pre-allocated and I will have 2Gb heap for streaming?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Should-I-
> decrease-the-taskmanager-memory-fraction-tp10475p12542.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Re: Should I decrease the taskmanager.memory.fraction ?

Posted by nragon <nu...@wedotechnologies.com>.
Like stephan said we don't need to worry about memory managment with
streaming? Meaning if I have 3 taskmanagers with 2Gb each, memory segments
won't be pre-allocated and I will have 2Gb heap for streaming?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475p12542.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Should I decrease the taskmanager.memory.fraction ?

Posted by Fabian Hueske <fh...@gmail.com>.
Yes, that's correct. Managed memory is currently only used for batch
processing.

For stream processing data is either kept on the JVM heap
(InMemoryStateBackend, FsStateBackend) or in RocksDB (RocksDBStateBackend).
There are plans to implement a state backend that serializes to to managed
memory (possibly spilling to disk if memory is exceeded).

2017-04-06 16:36 GMT+02:00 nragon <nu...@wedotechnologies.com>:

> Hi,
>
> Regarding this topic, does flink uses memory segments for window states?
> Is memory managment only "used at is full capacity" for batch?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Should-I-
> decrease-the-taskmanager-memory-fraction-tp10475p12540.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>