You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by TechnoMage <ml...@technomage.com> on 2018/04/11 21:04:35 UTC
State management and heap usage
I am pretty new to flink and have an initial streaming job working both locally and remotely. But, both ways if the data volume is too high it runs out of heap. I am using RichMapFunction to process multiple streams of data. I assumed Flink would manage keeping state in ram when possible, and spill to RocksDB when it exceeded heap.
Is this correct? If so are there configs I need to set to enable or tune this so it can run within a fixed memory size?
Michael
Re: State management and heap usage
Posted by TechnoMage <ml...@technomage.com>.
Thank you.
Michael
> On Apr 12, 2018, at 2:45 AM, Gary Yao <ga...@data-artisans.com> wrote:
>
> Hi Michael,
>
> You can configure the default state backend by setting state.backend in
> flink-conf.yaml, or you can configure it per job [1]. The default state backend
> is "jobmanager" (MemoryStateBackend), which stores state and checkpoints on the
> Java heap. RocksDB must be explicitly enabled, e.g., by setting state.backend to
> "rocksdb".
>
> Best,
> Gary
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#configuring-a-state-backend <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#configuring-a-state-backend>
>
> On Wed, Apr 11, 2018 at 11:04 PM, TechnoMage <mlatta@technomage.com <ma...@technomage.com>> wrote:
> I am pretty new to flink and have an initial streaming job working both locally and remotely. But, both ways if the data volume is too high it runs out of heap. I am using RichMapFunction to process multiple streams of data. I assumed Flink would manage keeping state in ram when possible, and spill to RocksDB when it exceeded heap.
>
> Is this correct? If so are there configs I need to set to enable or tune this so it can run within a fixed memory size?
>
> Michael
>
Re: State management and heap usage
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Michael,
You can configure the default state backend by setting state.backend in
flink-conf.yaml, or you can configure it per job [1]. The default state
backend
is "jobmanager" (MemoryStateBackend), which stores state and checkpoints on
the
Java heap. RocksDB must be explicitly enabled, e.g., by setting
state.backend to
"rocksdb".
Best,
Gary
[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#configuring-a-state-backend
On Wed, Apr 11, 2018 at 11:04 PM, TechnoMage <ml...@technomage.com> wrote:
> I am pretty new to flink and have an initial streaming job working both
> locally and remotely. But, both ways if the data volume is too high it
> runs out of heap. I am using RichMapFunction to process multiple streams
> of data. I assumed Flink would manage keeping state in ram when possible,
> and spill to RocksDB when it exceeded heap.
>
> Is this correct? If so are there configs I need to set to enable or tune
> this so it can run within a fixed memory size?
>
> Michael